In a recent post on its official engineering blog, Uber, disclosed its strategy to migrate the batch data analytics and machine learning (ML) training stack to Google Cloud Platform (GCP). Uber, runs one of the largest Hadoop installations in the world, managing over an exabyte of data across tens of thousands of servers in each of its two regions. The open-source data ecosystem, particularly Hadoop, has been the cornerstone of the data platform.
The strategic migration plan consists of two steps: Initial migration and leveraging Cloud-Native Services. Uber’s initial strategy involves leveraging GCP’s object store for data lake storage while migrating the rest of their data stack to GCP’s Infrastructure as a Service (IaaS). This approach allows for a swift migration with minimal disruption to the existing jobs and pipelines, as they can replicate the exact versions of their on-premises software stack, engines, and security model on IaaS. Following this phase, the Uber engineering team plans to gradually adopt GCP’s Platform as a Service (PaaS) offerings, such as Dataproc and BigQuery, to harness the elasticity and performance benefits of cloud-native services fully.
Once the initial migration is complete, the team will focus on integrating cloud-native services to maximize the data infrastructure’s performance and scalability. This phased approach ensures that Uber users, from dashboard owners to ML practitioners, experience a seamless transition without altering their existing workflows or services.
To ensure a smooth and efficient migration, the Uber team have established several guiding principles:
Minimize use disruption by moving the majority of the batch data stack onto cloud IaaS as-is; they aim to shield their users from any changes to their artifacts or services. Using well-known abstractions and open standards, they strive to make the migration as transparent as possible.
They will rely on a cloud storage connector that implements the Hadoop FileSystem interface to Google Cloud Storage, ensuring HDFS compatibility. By standardizing their Apache Hadoop HDFS clients, we will abstract the specifics of the on-premise HDFS implementation, allowing seamless integration with GCP’s storage layer.
The Uber team has developed data access proxies for Presto, Spark, and Hive that abstract the underlying physical compute clusters. These proxies will support the selective routing of test traffic to cloud-based clusters during the testing phase and fully route queries and jobs to the cloud stack during the full migration.
Utilizing Uber’s cloud-agnostic infrastructure. Uber existing container environment, computing platform, and deployment tools are built to be agnostic between cloud and on-premises. These platforms will enable to easily expand their batch data ecosystem microservices onto the cloud IaaS.
The team will build and enhance existing data management services to support selected and approved cloud services, ensuring robust data governance. The company aims to maintain the same levels of authorized access and security as on-premises, while supporting seamless user authentication against the object store data lake and other cloud services.
Pre and post-migration Uber’s batch data stack (source: Uber’s blog)
The Uber team focuses on bucket mapping and cloud resource layout for migration. Mapping HDFS files and directories to cloud objects in one or more buckets is critical. They need to apply IAM policies at varying levels of granularity, considering constraints on buckets and objects such as read/write throughput and IOPS throttling. The team aims to develop a mapping algorithm that satisfies these constraints and organizes data resources in an organization-centric hierarchical manner, improving data administration and management.
Security Integration is another workstream; adapting our existing Kerberos-based tokens and Hadoop Delegation tokens for cloud PaaS, particularly Google Cloud Storage (GCS), is essential. This workstream aims to support seamless user, group, and service account authentication and authorization, maintaining consistent access levels as on-premises.
The team also focuses on data replication. HiveSync, the permissions-aware bidirectional data replication service, allows Uber to operate in active-active mode. It extends HiveSync’s capabilities to replicate the on-premise data lake’s data to the cloud-based data lake and corresponding Hive Metastore. This includes an initial bulk migration and ongoing incremental updates until the cloud-based stack becomes the primary.
The last workstream is providing new YARN and Presto clusters on GCP Iaas. Uber data access proxies will route query and job traffic to these cloud-based clusters during the migration, ensuring a smooth transition.
Uber’s big data migration to Google Cloud anticipates challenges like performance differences in storage and unforeseen issues due to its legacy system. The team plans to address these by leveraging open-source tools, utilizing cloud elasticity for cost management, migrating non-core uses to dedicated storage, and proactively testing integrations and deprecating outdated practices.
CHICAGO (AP) — United Airlines has struck a deal with Elon Musk’s SpaceX to offer satellite-based Starlink WiFi service on flights within the next several years.
The airline said Friday the service will be free to passengers.
United said it will begin testing the service early next year and begin offering it on some flights by later in 2025.
Financial details of the deal were not disclosed.
The announcement comes as airlines rush to offer more amenities as a way to stand out when passengers pick a carrier for a trip. United’s goal is to make sitting on a plane pretty much like being on the ground when it comes to browsing the internet, streaming entertainment and playing games.
“Everything you can do on the ground, you’ll soon be able to do on board a United plane at 35,000 feet, just about anywhere in the world,” CEO Scott Kirby said in announcing the deal.
The airline says Starlink will allow passengers to get internet access even over oceans and polar regions where traditional cell or Wi-Fi signals may be weak or missing.
Sony has made it easy for Canadian consumers to preorder the PlayStation 5 Pro in Canada directly from PlayStation’s official website. Here’s how:
Visit the Official Website: Go to direct.playstation.com and navigate to the PS5 Pro section once preorders go live on September 26, 2024.
Create or Log in to Your PlayStation Account: If you don’t have a PlayStation account, you will need to create one. Existing users can simply log in to proceed.
Place Your Preorder: Once logged in, follow the instructions to preorder your PS5 Pro. Ensure you have a valid payment method ready and double-check your shipping information for accuracy.
Preorder Through Major Canadian Retailers
While preordering directly from PlayStation is a popular option, you can also secure your PS5 Pro through trusted Canadian retailers. These retailers are expected to offer preorders on or after September 26:
Best Buy Canada
Walmart Canada
EB Games (GameStop)
Amazon Canada
The Source
Steps to Preorder via Canadian Retailers:
Visit Retailer Websites: Search for “PlayStation 5 Pro” on the website of your preferred retailer starting on September 26.
Create or Log in to Your Account: If you’re shopping online, having an account with the retailer can speed up the preorder process.
Preorder in Store: For those who prefer in-person shopping, check with local stores regarding availability and preorder policies.
3. Sign Up for Notifications
Many retailers and websites offer the option to sign up for notifications when the preorder goes live. If you’re worried about missing out due to high demand, this can be a useful option.
Visit Retailer Sites: Look for a “Notify Me” or “Email Alerts” option and enter your email to stay informed.
Use PlayStation Alerts: Sign up for notifications directly through Sony to be one of the first to know when preorders are available.
4. Prepare for High Demand
Preordering the PS5 Pro is expected to be competitive, with high demand likely to result in quick sellouts, just as with the initial release of the original PS5. To maximize your chances of securing a preorder:
Act Quickly: Be prepared to place your order as soon as preorders open. Timing is key, as stock can run out within minutes.
Double-Check Payment Information: Ensure your credit card or payment method is ready to go. Any delays during the checkout process could result in losing your spot.
Stay Informed: Monitor PlayStation and retailer websites for updates on restocks or additional preorder windows.
Final Thoughts
The PlayStation 5 Pro is set to take gaming to the next level with its enhanced performance, graphics, and new features. Canadian gamers should be ready to act fast when preorders open on September 26, 2024, to secure their console ahead of the holiday season. Whether you choose to preorder through PlayStation’s official website or your preferred retailer, following the steps outlined above will help ensure a smooth and successful preorder experience.
For more details on the PS5 Pro and to preorder, visit direct.playstation.com or stay tuned to updates from major Canadian retailers.
Since the PlayStation 5 (PS5) launched four years ago, PlayStation has continuously evolved to meet the demands of its players. Today, we are excited to announce the next step in this journey: the PlayStation 5 Pro. Designed for the most dedicated players and game creators, the PS5 Pro brings groundbreaking advancements in gaming hardware, raising the bar for what’s possible.
Key Features of the PS5 Pro
The PS5 Pro comes equipped with several key performance enhancements, addressing the requests of gamers for smoother, higher-quality graphics at a consistent 60 frames per second (FPS). The console’s standout features include:
Upgraded GPU: The PS5 Pro’s GPU boasts 67% more Compute Units than the current PS5, combined with 28% faster memory. This allows for up to 45% faster rendering speeds, ensuring a smoother gaming experience.
Advanced Ray Tracing: Ray tracing capabilities have been significantly enhanced, with reflections and refractions of light being processed at double or triple the speed of the current PS5, creating more dynamic visuals.
AI-Driven Upscaling: Introducing PlayStation Spectral Super Resolution, an AI-based upscaling technology that adds extraordinary detail to images, resulting in sharper image clarity.
Backward Compatibility & Game Boost: More than 8,500 PS4 games playable on PS5 Pro will benefit from PS5 Pro Game Boost, stabilizing or enhancing performance. PS4 games will also see improved resolution on select titles.
VRR & 8K Support: The PS5 Pro supports Variable Refresh Rate (VRR) and 8K gaming for the ultimate visual experience, while also launching with the latest wireless technology, Wi-Fi 7, in supported regions.
Optimized Games & Patches
Game creators have quickly embraced the new technology that comes with the PS5 Pro. Many games will receive free updates to take full advantage of the console’s new features, labeled as PS5 Pro Enhanced. Some of the highly anticipated titles include:
Alan Wake 2
Assassin’s Creed: Shadows
Demon’s Souls
Dragon’s Dogma 2
Final Fantasy 7 Rebirth
Gran Turismo 7
Marvel’s Spider-Man 2
Ratchet & Clank: Rift Apart
Horizon Forbidden West
These updates will allow players to experience their favorite games at a higher fidelity, taking full advantage of the console’s improved graphics and performance.
Design & Compatibility
Maintaining consistency within the PS5 family, the PS5 Pro retains the same height and width as the original PS5 model. Players will also have the option to add an Ultra HD Blu-ray Disc Drive or swap console covers when available.
Additionally, the PS5 Pro is fully compatible with all existing PS5 accessories, including the PlayStation VR2, DualSense Edge, Pulse Elite, and Access controller. This ensures seamless integration into your current gaming setup.
Pricing & Availability
The PS5 Pro will be available starting November 7, 2024, at a manufacturer’s suggested retail price (MSRP) of:
$699.99 USD
$949.99 CAD
£699.99 GBP
€799.99 EUR
¥119,980 JPY
Each PS5 Pro comes with a 2TB SSD, a DualSense wireless controller, and a copy of Astro’s Playroom pre-installed. Pre-orders begin on September 26, 2024, and the console will be available at participating retailers and directly from PlayStation via direct.playstation.com.
The launch of the PS5 Pro marks a new chapter in PlayStation’s commitment to delivering cutting-edge gaming experiences. Whether players choose the standard PS5 or the PS5 Pro, PlayStation aims to provide the best possible gaming experience for everyone.
Preorder your PS5 Pro and step into the next generation of gaming this holiday season.
/filters:no_upscale()/news/2024/06/uber-bigdata-migration-gcp/en/resources/1Untitled%20(1)-1719563126326.jpg)
