We discuss common use cases and challenges for copying data between S3 buckets and S3-compatible object storage services. We share our experience building an open source Node.js CLI tool called S3-Migrate to efficiently migrate data with separate source and destination credentials. We cover performance considerations like streaming, chunk sizes, concurrency and parallelism.AWS Bites is brought to you in association with fourTheorem. If you need a friendly partner to support you and work with you to de-risk any AWS migration or development project, check them out at fourtheorem.comIn this episode, we mentioned the following resources:s3-migrate CLI tool: https://github.com/lmammino/s3-migrateDigitalOcean Spaces Object Storage: https://docs.digitalocean.com/products/spaces/Cloudflare R2: https://www.cloudflare.com/en-gb/developer-platform/products/r2/Backblaze B2: https://www.backblaze.com/cloud-storageWasabi Cloud Storage: https://wasabi.com/Linode / Akamai Object Storage: https://www.linode.com/products/object-storage/MinIO (Self-hosted S3-compatible storage): https://min.io/Basecamp / Hey’s move away from S3: https://world.hey.com/dhh/it-s-five-grand-a-day-to-miss-our-s3-exit-b8293563AWS re:Post - How to move objects between S3 buckets: https://repost.aws/knowledge-center/move-objects-s3-bucketAWS Labs - Utility for S3 Migration: https://github.com/awslabs/utility-for-s3-migrations3s3mirror (Java-based tool): https://github.com/cobbzilla/s3s3mirrorrclone S3 Support: https://rclone.org/s3/knox-copy (Ruby-based, deprecated): https://github.com/goodeggs/knox-copyFlexify.io (paid cloud migration service): https://flexify.io/Do you have any AWS questions you would like us to address?Leave a comment here or connect with us on X/Twitter, BlueSky or LinkedIn:- https://twitter.com/eoins | https://bsky.app/profile/eoin.sh | https://www.linkedin.com/in/eoins/- https://twitter.com/loige | https://bsky.app/profile/loige.co | https://www.linkedin.com/in/lucianomammino/
--------
30:44
141. Step Functions with JSONata and Variables
In this episode, we provide an overview of AWS Step Functions and dive deep into the powerful new JSONata and variables features. We explain how JSONata allows complex JSON transformations without custom Lambda functions, enabling more serverless workflows. The variables feature also helps avoid the previous 256KB state size limit. We share examples from real projects showing how these features simplify workflows, reduce costs and enable new use cases.AWS Bites is brought to you in association with fourTheorem. If you need a friendly partner to support you and work with you to de-risk any AWS migration or development project, check them out at fourtheorem.comIn this episode, we mentioned the following resources:JSONata and variables official launch post: https://aws.amazon.com/blogs/compute/simplifying-developer-experience-with-variables-and-jsonata-in-aws-step-functions/JSONata exerciser: https://try.jsonata.org/Stedi JSONata playground: https://www.stedi.com/jsonata/playgroundEpisode 103: Building GenAI Features with Bedrock https://awsbites.com/103-building-genai-features-with-bedrock/Episode 63: How to automate transcripts with Amazon Transcribe and OpenAI Whisper https://awsbites.com/63-how-to-automate-transcripts-with-amazon-transcribe-and-openai-whisper/ Do you have any AWS questions you would like us to address?Leave a comment here or connect with us on X/Twitter, BlueSky or LinkedIn:- https://twitter.com/eoins | https://bsky.app/profile/eoin.sh | https://www.linkedin.com/in/eoins/- https://twitter.com/loige | https://bsky.app/profile/loige.co | https://www.linkedin.com/in/lucianomammino/
--------
15:43
140. DuckDB Meets AWS: A Match Made in Cloud
In this episode, we explore DuckDB, an open-source analytical database known for its speed and simplicity. Discover how DuckDB stands out in various applications and compare it to other tools like SQLite, Athena, Pandas, and Polars. We also demonstrate integrating DuckDB with AWS Lambda and Step Functions for serverless analytics.AWS Bites is brought to you by fourTheorem. If you are looking for a partner to architect, develop and modernise on AWS, give fourTheorem a call. Check out fourtheorem.comIn this episode, we mentioned the following resources: Our `duck-query-lambda`, A Lambda runtime for DuckDB queries: https://github.com/fourTheorem/duck-query-lambda DuckDB's official website: https://duckdb.org/ LibSQL: https://github.com/tursodatabase/libsql Do you have any AWS questions you would like us to address?Leave a comment here or connect with us on X/Twitter, BlueSky or LinkedIn:- https://twitter.com/eoins | https://bsky.app/profile/eoin.sh | https://www.linkedin.com/in/eoins/- https://twitter.com/loige | https://bsky.app/profile/loige.co | https://www.linkedin.com/in/lucianomammino/
--------
17:38
139. Building Great APIs with Powertools
In this episode, we discuss using AWS Lambda Powertools for Python to build serverless REST APIs with AWS Lambda. We cover the benefits of using Powertools for routing, validation, OpenAPI support, and more. Powertools provides an excellent framework for building APIs while maintaining Lambda best practices.In this episode, we mentioned the following resources: AWS Bites 41. How can Middy make writing Lambda functions easier? - https://awsbites.com/41-how-can-middy-make-writing-lambda-functions-easier AWS Bites 120. Lambda Best Practices - https://awsbites.com/120-lambda-best-practices/ REST API - Powertools for AWS Lambda (Python) - https://docs.powertools.aws.dev/lambda/python/latest/core/event_handler/api_gateway/ Hono - https://hono.dev/ Fastify - https://fastify.dev/ Axum - https://github.com/tokio-rs/axum FastAPI - https://fastapi.tiangolo.com/Do you have any AWS questions you would like us to address?Leave a comment here or connect with us on BlueSky or LinkedIn: https://bsky.app/profile/eoin.sh | https://www.linkedin.com/in/eoins/ https://bsky.app/profile/loige.co | https://www.linkedin.com/in/lucianomammino/
--------
24:32
138. How Do You Become A Cloud Architect?
Ready to take your tech career to the cloud and build those awe-inspiring systems you see? Then you're in the right place. This episode of AWS Bites is your blueprint for becoming a successful cloud architect. We're not just going to talk about it; we'll show you what worked for us, sharing the critical skills you need, and a practical path to build your expertise. Whether you're a beginner or looking to take the next step, join us as we equip you with the knowledge and tools to make your mark as a cloud architect!
In this episode, we mentioned the following resources:
Google Cloud Architecture Definition: https://cloud.google.com/learn/what-is-cloud-architecture
Market data about the Cloud Professional Services market: https://www.gminsights.com/industry-analysis/cloud-professional-services-market
EP 91 - Our Journeys into Software and AWS: https://awsbites.com/91-our-journeys-into-software-and-aws/
AWS Well-Architected Framework: https://aws.amazon.com/architecture/well-architected/
EP 68 - Are you well architected?: https://awsbites.com/68-are-you-well-architected/
Cloud Design Patterns: https://learn.microsoft.com/en-us/azure/architecture/patterns/
The art of scalability (book): https://www.amazon.com/Art-Scalability-Architecture-Organizations-Enterprise/dp/0134032802
Enterprise integration patterns (book): https://www.amazon.com/Enterprise-Integration-Patterns-Designing-Deploying/dp/0321200683/
Designing Data-Intensive Applications (book): https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321
AWS Networking Essentials (free guide): https://aws.amazon.com/getting-started/aws-networking-essentials/
Docker Curriculum (free): https://docker-curriculum.com/
How Linux works (book): https://www.amazon.com/How-Linux-Works-Brian-Ward/dp/1718500408/
Exercism coding challenges: https://exercism.org/
The tangled web (book): https://www.amazon.co.uk/Tangled-Web-Securing-Modern-Applications/dp/1593273886
Low Level YouTube Channel: https://www.youtube.com/lowlevellearning
AWS - Best Practices for Security, Identity, & Compliance: https://aws.amazon.com/architecture/security-identity-compliance/
Supercommunicators (book): https://www.amazon.com/Supercommunicators-Unlock-Secret-Language-Connection/dp/0593862066/
An Elegant Puzzle (book): https://www.amazon.com/Elegant-Puzzle-Systems-Engineering-Management/dp/1732265186/
Staff Engineer (book): https://www.amazon.com/Staff-Engineer-Leadership-beyond-management/dp/1736417916/
EP 58 - What can kitties teach us about AWS: https://awsbites.com/58-what-can-kitties-teach-us-about-aws/
AWS User Groups: https://aws.amazon.com/developer/community/usergroups/
Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on X/Twitter:
- https://twitter.com/eoins
- https://twitter.com/loige
AWS Bites is the weekly show where we answer questions about AWS! This show is brought to you be Eoin Shanaghy and Luciano Mammino, certified AWS experts.
UK