S3 to HF Bucket Importer

1

Connect to S3

Provide your AWS credentials so we can list your S3 bucket contents. Credentials are used in your browser to browse files, then passed as encrypted secrets to the import Job. They are not stored beyond the lifetime of the Job.

AWS Access Key ID (optional for public buckets)

Your IAM access key ID, starting with AKIA. You can create one in the AWS IAM Console under "Access keys". Leave empty for public S3 buckets that allow anonymous read access.

AWS Secret Access Key (optional for public buckets)

The secret key paired with your Access Key ID. It was shown only once when you created the access key. This value is sent to the HF Job as an encrypted secret and is not persisted beyond the Job's lifetime. Leave empty for public S3 buckets.

S3 Bucket Name *

The name of your S3 bucket (not the full ARN or URL). For example, if your bucket URL is s3://my-data-bucket, enter my-data-bucket.

AWS Region (for AWS S3)

The AWS region where your S3 bucket is hosted, e.g. us-east-1, eu-west-1, ap-southeast-1. You can find this in the AWS Console next to your bucket name, or in the bucket's Properties tab. For non-AWS services, use the Endpoint URL field below instead.

Endpoint URL (for non-AWS services)

For S3-compatible services like MinIO, DigitalOcean Spaces, Cloudflare R2, Scaleway, OVHcloud, etc. When set, this replaces the AWS region — the two fields are mutually exclusive. Leave empty for standard AWS S3 (the region above will be used).

Source Prefix (optional)

Filter to a subfolder within your bucket. Only files under this path will be listed and imported. For example, data/exports/ will only show files starting with that path. Leave empty to browse the entire bucket.

2

Browse & Select Files

Selected: 0 files Size: 0 B Total: 0 files, 0 B

3

Configure Import

Choose where to import your files on Hugging Face. The bucket will be created automatically if it doesn't exist yet. The import runs as an HF Job on Hugging Face infrastructure — your browser does not need to stay open.

Destination Bucket *

The HF Storage Bucket to import files into. You can use:
• bucket-name — creates under your username (username/bucket-name)
• org/bucket-name — creates under a specific organization
The bucket will be created for you if it doesn't exist yet.

Destination Prefix (optional)

Place imported files under this path within the bucket. For example, imported/2024/ will import file.txt as imported/2024/file.txt in the bucket. Leave empty to import at the bucket root.

Create bucket as private

Private buckets are only accessible to you (and your organization members). Uncheck to make the bucket publicly readable. This only applies when creating a new bucket — existing buckets keep their current visibility.

Hardware

The machine type for the import Job. S3-to-bucket transfer is network-bound, so only CPU options are shown. Larger machines provide more vCPUs, memory, and network bandwidth for faster imports. The number of workers is automatically set to match the vCPU count. You are billed per second of Job runtime.

Workers

Number of parallel download threads. Higher values speed up the transfer when importing many small files. For large files (100MB+), 2-4 workers is usually sufficient. Max: 16.

Buffer Size

Maximum local disk space the Job can use for buffering files during transfer. Files are downloaded from S3 to a temporary buffer, then uploaded to the HF bucket. Any single file larger than this limit cannot be imported. Use units like 10g, 500m, 1t (k/m/g/t/p). Job machines have ~1-1.6 TB of disk depending on hardware, so values up to 1t are reasonable.

Timeout

Maximum time the Job is allowed to run before being automatically stopped. Choose a value that covers your expected transfer time. For large datasets (100GB+), pick 6h or more. You are only billed for actual runtime, not the full timeout.

🚀

Scheduling

Import Job Submitted

View Job on Hugging Face →

View Bucket →

S3 → HF Buckets

Secure

Server-side

Fast

Connect to S3

Browse & Select Files

Configure Import

Import Job Submitted