yacloud/shadowmire
1
0
Fork 0
mirror of https://github.com/taoky/shadowmire.git synced 2025-07-08 17:32:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
84 Commits 3 Branches 0 Tags
Commit Graph

74 Commits

Author SHA1 Message Date
taoky
40c9edba50 Add generate pathcache as Step 3 
This helps speed up check_and_update(), when --compare-size is NOT set.
 2024-09-04 00:18:08 +08:00
taoky
7c963b00b1 Check package size in do_update() 
This helps make --compare-size really work.
 2024-09-03 23:10:51 +08:00
taoky
a5dbd61e95 Replace glob by fast_iterdir in verification step 4 
To avoid stat() every files in package/.
Related: #4
 2024-09-03 22:47:43 +08:00
taoky
6e4aa61119 Add fast_readall to save some syscalls 
Related: #4
Co-authored-by: luvletter2333 <luvletter2333@gmail.com>
 2024-09-03 17:03:55 +08:00
taoky
cd40fc1f40 DRY in futures exception handling 2024-08-27 21:39:35 +08:00
taoky
99fabce7b8 Parallelize verification step 4 
Related: #4
 2024-08-27 20:33:36 +08:00
taoky
9e57eb6f46 Optimize get_existing_hrefs for fast path (json file exists) 
Related: #4
 2024-08-27 19:20:20 +08:00
taoky
799a477336 Parallelize verification step 3 with threads 
Related: #4
 2024-08-27 19:06:05 +08:00
taoky
d64ea61be7 Only do one stat() for packages when sync-packages is on in verification step 3 
Related: #4
 2024-08-27 18:46:19 +08:00
taoky
f4927fdb0d Save a bit syscalls in check_and_update() 
Related: #4
 2024-08-27 18:26:02 +08:00
taoky
bc6159807d iterdir() optimization 
Related: #4
 2024-08-27 16:36:37 +08:00
taoky
17d48f2456 Add type parameter for package_names list 2024-08-24 17:41:31 +08:00
taoky
972c94d35b Add MAX_DELETION "safety net" 2024-08-24 17:39:06 +08:00
taoky
0a6a958e49 Fix getting last serial in SyncPlainHTTP 2024-08-11 05:07:44 +08:00
taoky
d565ac388b Fix urljoin usage in SyncPlainHTTP 2024-08-11 04:46:47 +08:00
taoky
b9c6ac5c62 Add verification for v1_html and symlink 2024-08-10 19:39:00 +08:00
taoky
b5fca47101 Make simple/<package>/index.html symlink to index.v1_html 
Closes: #1
 2024-08-10 19:35:04 +08:00
taoky
a913965c22 Fix logging in genlocal 2024-08-08 18:16:13 +08:00
taoky
4e20615ed6 Fix logging not showing 2024-08-08 16:58:06 +08:00
Shengqi Chen
4d509dfb5e chore: add more logging output 
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
 2024-08-08 16:40:07 +08:00
Shengqi Chen
d09c3a8ed1 chore: adjust default logging format 
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
 2024-08-08 16:40:07 +08:00
Shengqi Chen
b6071ec2e9 chore: seprate third-party imports 
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
 2024-08-08 16:40:07 +08:00
taoky
f1a2a023de Some linting fixes 2024-08-08 14:52:39 +08:00
taoky
4d602a3649 Add --use-pypi-index 2024-08-08 14:50:22 +08:00
taoky
008617cccd Set repo with argument/config (no longer use env) 2024-08-08 14:33:34 +08:00
taoky
1cfb6c8a99 More normpath() to replace resolve() 2024-08-08 06:28:25 +08:00
taoky
c120c2c747 Fix step 3 syncing removed packages in verify, and performance improvement in check_and_update() 2024-08-08 06:24:20 +08:00
taoky
5637860cde Avoid unnecessary fs syscalls in verify step 4 2024-08-08 06:09:37 +08:00
taoky
e9781e7cd9 Add tqdm for step 4 in verify 2024-08-08 01:09:42 +08:00
taoky
9dbc320a1e Handle cases where size is not available from JSON 2024-08-06 23:45:11 +08:00
taoky
dad48fc8a9 Defer package removal in verify 2024-08-06 23:28:02 +08:00
taoky
6102de1bf7 Save remote_excluded.json to basedir 2024-08-06 23:00:08 +08:00
taoky
5585de30ce Change --not-remove-not-in-local to --remove-not-in-local 2024-08-06 22:09:36 +08:00
taoky
baef090078 Print package number to update/remove in verify 2024-08-06 22:06:30 +08:00
taoky
e5c542c718 Show remote package numbers, and also export remote to JSON file after exclusion 2024-08-06 22:02:41 +08:00
taoky
79a2dec5f5 Add --compare-size for verify 2024-08-06 21:57:53 +08:00
taoky
2bdb9355cf Add --no-remove-not-in-local for verify 2024-08-06 21:08:54 +08:00
taoky
fcb8f8bfee Log printing and debugging 
Co-authored-by: Harry-Chen <i@harrychen.xyz>
 2024-08-06 21:03:05 +08:00
taoky
aa4ae7e477 Add some type hints for mypy strict mode 2024-08-06 18:39:23 +08:00
taoky
6751bc93cb Add prerelease filter support in verify 2024-08-06 18:23:07 +08:00
taoky
33e9746a46 Implement prerelease filter in SyncPlainHTTP 2024-08-06 18:17:18 +08:00
taoky
231b891565 Write original meta to json/ 2024-08-06 18:03:26 +08:00
taoky
b97754ce34 Update check_and_update() to check both json and html 2024-08-06 18:00:27 +08:00
taoky
c8a11d3d1f Config file support and other fixes 
- Write JSON every 100 iterations instead of 1000 iterations
- Write plan.json with indent=2
 2024-08-06 14:41:51 +08:00
taoky
a6358aad8d Add logger.info to verify 2024-08-06 13:22:08 +08:00
taoky
18d6847f76 Misc fixes 
- Show speed for genlocal
- Fix get_existing_hrefs() behavior
- Add indent for dump_json()
 2024-08-06 13:11:56 +08:00
taoky
c613d21fa1 Fix genlocal behavior 2024-08-06 12:58:10 +08:00
taoky
70a99b8594 Add json/<package_name> support and bug fixes 2024-08-06 03:06:21 +08:00
taoky
2aa8c9ae97 Adjust verify behavior 
Verifying is heavily network-bound if we need to update EVERY package, so let's just check if local is consist instead of making sure everything is up-to-date.
 2024-08-05 22:04:33 +08:00
taoky
bede75051e Overwrite index after package sync in SyncPlainHTTP do_update 2024-08-05 20:22:03 +08:00