51 Commits

Author	SHA1	Message	Date
taoky	4d602a3649	Add --use-pypi-index	2024-08-08 14:50:22 +08:00
taoky	008617cccd	Set repo with argument/config (no longer use env)	2024-08-08 14:33:34 +08:00
taoky	1cfb6c8a99	More normpath() to replace resolve()	2024-08-08 06:28:25 +08:00
taoky	c120c2c747	Fix step 3 syncing removed packages in verify, and performance improvement in check_and_update()	2024-08-08 06:24:20 +08:00
taoky	5637860cde	Avoid unnecessary fs syscalls in verify step 4	2024-08-08 06:09:37 +08:00
taoky	e9781e7cd9	Add tqdm for step 4 in verify	2024-08-08 01:09:42 +08:00
taoky	9dbc320a1e	Handle cases where size is not available from JSON	2024-08-06 23:45:11 +08:00
taoky	dad48fc8a9	Defer package removal in verify	2024-08-06 23:28:02 +08:00
taoky	6102de1bf7	Save remote_excluded.json to basedir	2024-08-06 23:00:08 +08:00
taoky	5585de30ce	Change --not-remove-not-in-local to --remove-not-in-local	2024-08-06 22:09:36 +08:00
taoky	baef090078	Print package number to update/remove in verify	2024-08-06 22:06:30 +08:00
taoky	e5c542c718	Show remote package numbers, and also export remote to JSON file after exclusion	2024-08-06 22:02:41 +08:00
taoky	79a2dec5f5	Add --compare-size for verify	2024-08-06 21:57:53 +08:00
taoky	2bdb9355cf	Add --no-remove-not-in-local for verify	2024-08-06 21:08:54 +08:00
taoky	fcb8f8bfee	Log printing and debugging ... Co-authored-by: Harry-Chen <i@harrychen.xyz>	2024-08-06 21:03:05 +08:00
taoky	aa4ae7e477	Add some type hints for mypy strict mode	2024-08-06 18:39:23 +08:00
taoky	6751bc93cb	Add prerelease filter support in verify	2024-08-06 18:23:07 +08:00
taoky	33e9746a46	Implement prerelease filter in SyncPlainHTTP	2024-08-06 18:17:18 +08:00
taoky	231b891565	Write original meta to json/	2024-08-06 18:03:26 +08:00
taoky	b97754ce34	Update check_and_update() to check both json and html	2024-08-06 18:00:27 +08:00
taoky	c8a11d3d1f	Config file support and other fixes ... - Write JSON every 100 iterations instead of 1000 iterations - Write plan.json with indent=2	2024-08-06 14:41:51 +08:00
taoky	a6358aad8d	Add logger.info to verify	2024-08-06 13:22:08 +08:00
taoky	18d6847f76	Misc fixes ... - Show speed for genlocal - Fix get_existing_hrefs() behavior - Add indent for dump_json()	2024-08-06 13:11:56 +08:00
taoky	c613d21fa1	Fix genlocal behavior	2024-08-06 12:58:10 +08:00
taoky	70a99b8594	Add json/<package_name> support and bug fixes	2024-08-06 03:06:21 +08:00
taoky	2aa8c9ae97	Adjust verify behavior ... Verifying is heavily network-bound if we need to update EVERY package, so let's just check if local is consist instead of making sure everything is up-to-date.	2024-08-05 22:04:33 +08:00
taoky	bede75051e	Overwrite index after package sync in SyncPlainHTTP do_update	2024-08-05 20:22:03 +08:00
taoky	7df08fcbde	Call do_remove for nonexist package metadata in do_update	2024-08-05 20:19:43 +08:00
taoky	6f76afcdcc	Fix incorrect prerelease exclude behavior	2024-08-04 18:50:54 +08:00
taoky	649d14c4fd	Remove invalid local ones when planning	2024-08-04 17:14:49 +08:00
taoky	9008d2cbce	Fix invalid package skipping	2024-08-04 17:09:58 +08:00
taoky	0f97cb7dbd	Put serial -1 handling at LocalVersionKV	2024-08-04 16:51:23 +08:00
taoky	d92d2079bd	Mark not-found packages serial as -1 ... To speed up index updating.	2024-08-04 16:46:03 +08:00
taoky	7d4aef2f8a	Add do_update/do_remove command	2024-08-04 16:35:43 +08:00
taoky	5dc892c0b3	Add prerelease exclusion	2024-08-04 16:20:08 +08:00
taoky	c695419700	Use click to replace argparse ... Make it simpler and easier to maintain command line arguments.	2024-08-04 16:01:13 +08:00
taoky	8d2460cf73	Add shebang	2024-08-03 02:46:13 +08:00
taoky	ada070d10e	Add retry for requests.Session	2024-08-03 02:35:42 +08:00
taoky	9ecdfee147	Add some docs	2024-08-02 22:36:38 +08:00
taoky	af49fb183f	Add exclude func (for testing), and fix unreferenced files cleanup	2024-08-02 21:40:24 +08:00
taoky	103252ad14	Remove unreferenced files in verify subcommand	2024-08-02 20:47:01 +08:00
taoky	ead0f42e2e	Add worker number warning	2024-08-02 20:28:21 +08:00
taoky	1281e6950d	Dump local JSON when finalizing	2024-08-02 20:21:27 +08:00
taoky	52ceabcdd4	SyncPlainHTTP Packages downloading support	2024-08-02 20:20:25 +08:00
taoky	a4574ecaeb	PyPI Packages downloading support	2024-08-02 19:57:08 +08:00
taoky	ad57590bf0	Add threading for updating packages	2024-08-02 16:56:23 +08:00
taoky	889ed86497	Adjust verify behavior	2024-08-02 16:14:10 +08:00
taoky	b0187e947f	Formatting and linting	2024-08-02 16:01:59 +08:00
taoky	18a3ebbe03	Add local db	2024-08-02 15:48:55 +08:00
taoky	577d021d70	Metadata sync PoC	2024-08-02 06:35:00 +08:00