Improve VTDR Figshare download workflow with logging, file verification, duplicate-file detection, and resumable downloads

[Jiren1247]

Improve the VTDR Ingest and Publication Figshare download workflows to provide stronger validation before datasets proceed to bagging and preservation.

The workflow should verify that all expected Figshare files were downloaded, identify duplicated files within the dataset, allow data curators to review validation warnings before proceeding, preserve the original file structure, support rerunning interrupted downloads without unnecessarily downloading existing files again, and save workflow output to a log file.

Download validation should occur before the dataset proceeds to the transfer/bagging stage.

• Write workflow output to a log file
• Print the Figshare file list
• Check the file count after download
• Compare expected and downloaded file names
• Pause if the file verification does not match Would you like to proceed? (yes/no):
• Detect duplicated files in the dataset
• Print duplicated file groups
• Do not download valid existing files again
  File already exists and checksum matches. Skipping download: acousticindex_BOH.csv
• Detect incomplete/corrupted existing files
• Preserve the original Figshare file/folder structure
• Apply validation to both Ingest and Publication downloads

[Jiren1247]

Document Git Push Authentication and Secret Protection Troubleshooting
Add troubleshooting documentation for Git push failures encountered during the VTDR development workflow, particularly authentication issues and accidental inclusion of credential files in Git commit history.

1. GitHub Authentication Failure

If git push fails when authenticating through either SSH or a Personal Access Token (PAT), first close the current terminal session and open a new terminal before retrying.

This can help clear stale environment variables or temporary authentication settings, particularly when Git/VS Code is referencing an invalid askpass path.

For example:

fatal: cannot exec '.../Visual Studio Code.app/.../askpass.sh':
No such file or directory

After opening a new terminal, verify the repository remote:

git remote -v

For an HTTPS remote:

https://github.com/VTUL/VTDR_RepositoryServices.git

authenticate using:

GitHub username
Personal Access Token instead of the GitHub account password

For an SSH remote, verify SSH authentication with:

ssh -T git@github.com
2. Check for client_secret.json in Commit History

If authentication succeeds but the push is still rejected, check the GitHub error carefully for Push Protection or secret-scanning violations.

For example:

GH013: Repository rule violations found

GITHUB PUSH PROTECTION

Push cannot contain secrets

Google Cloud Service Account Credentials

path: Figshare-APTrust/client_secret.json

The client_secret.json file must not be committed to the repository.

Add it to the repository .gitignore:

/Figshare-APTrust/client_secret.json

If the file is already tracked by Git, remove it from Git tracking while keeping the local file:

git rm --cached Figshare-APTrust/client_secret.json
git add .gitignore

3. Check Whether the Secret Exists in Commit History

Adding the file to .gitignore does not remove it from existing commits.

Check the local commits that have not yet been pushed:

git log --oneline origin/<branch-name>..HEAD
Also check the history of the secret file:

git log --all --oneline -- Figshare-APTrust/client_secret.json
If client_secret.json appears in an unpushed commit, the commit history must be cleaned before pushing.

For a small number of local, unpushed commits, reset them while preserving the changes:

git reset --soft origin/<branch-name>

Verify that the secret file is no longer tracked:

git ls-files Figshare-APTrust/client_secret.json

This command should return no output.

Then recreate a clean commit:

git add .
git commit -m "<commit message>"

Check the new commit before pushing:

git show --stat --oneline HEAD

Confirm that Figshare-APTrust/client_secret.json is not included.

Finally:

git push