19/20. AI Supercluster: Site Selection for City-Scale Computing

Introduction

Building a city-scale supercluster equipped with over 10,000 GPUs is an ambitious endeavor that requires careful planning and consideration. Site selection is one of the most key components of this process, with far-reaching implications for the project's success. The location affects everything from cooling efficiency and power supply stability to compliance with data regulations and environmental sustainability. This article explores the multi-faceted factors that datacenter engineers and planners must consider when selecting a site for a supercluster, highlighting geographical, environmental, infrastructural, regulatory, and economic aspects.

1. Key Geographical and Environmental Factors to Consider

Selecting the right geographical location is fundamental to ensuring the reliability and cost-effectiveness of a datacenter. By thoroughly assessing environmental conditions, planners can make informed decisions that support operational stability.

Climate Considerations (Cooling Costs)

Countries like Finland and Sweden are popular for datacenters because of their naturally cool climates, which help reduce cooling costs.

For instance, Google's datacenter in Hamina, Finland, leverages both the cool climate and access to seawater for efficient cooling. These natural conditions significantly cut down on the energy required for traditional mechanical cooling systems, making them highly efficient from both a cost and operational standpoint.

Soil Conditions (Structural Costs)

The type of soil and the region's geology also affect the construction of a supercluster. For example, Iowa is preferred due to its stable soil conditions, which reduce the need for extensive foundational reinforcements, making it a cost-effective location for datacenter construction. Sites with stable, compact soil provide a solid foundation, reducing the need for extensive foundational reinforcements. These considerations tie into the structural planning and floor loading assessments discussed in Article 18, where datacenter build-out requires robust support to handle the weight of heavy racks and equipment.

Natural Disaster Risk Level

One of the first considerations for selecting a supercluster site is the local climate and susceptibility to natural disasters. Several regions already host key datacenters due to favorable characteristics:

• Seismic Stability: Locations like Iowa in the United States are preferred for their low risk of seismic activity and stable geology. This stability makes Iowa attractive to companies like Facebook and Microsoft, which have established key datacenters there. When dealing with thousands of GPUs, even minor disruptions can lead to key data loss or downtime, making seismic stability a key factor in site selection.

• Elevation: Oregon in the United States is home to several datacenters, including those of Amazon and Meta, partly due to its suitable elevation, which reduces flood risks and offers access to hydroelectric power. Flood zones are a key consideration, as the damage from flooding could incapacitate the datacenter. Elevation and appropriate site grading ensure that the site remains operational even during extreme weather events.

Examples of mitigation strategies for regions exposed to natural disaster risks, include

• Datacenters in Japan have incorporated advanced seismic technologies due to the country’s vulnerability to earthquakes.

• The Netherlands has extensive experience with flood control, and their datacenters often implement advanced flood prevention measures.

Summary of Geographical and Environmental Considerations

The geographical factors, such as climate, disaster preparedness, and soil stability, directly influence the overall sustainability and cost efficiency of a datacenter. Connecting these considerations to the larger themes of operational stability and system resilience helps to underscore the key role of these factors in ensuring a reliable infrastructure.

2. Securing Megawatt Levels of Power

Power Contracts

Securing power contracts is a critical and often time-consuming aspect of supercluster planning. Obtaining contracts for 100+ megawatt-levels, equivalent to a small city's power draw, can be a major bottleneck in datacenter deployment. The timeline for these contracts often depends on the capacity of the regional grid and the willingness of utilities to upgrade infrastructure to meet demand. In many cases, negotiating and establishing power contracts takes longer than the physical build-out of the datacenter itself.

Determining the timeline and feasibility of expanding or upgrading power grids is key in site selection. New substations or grid upgrades may require several years, significantly impacting project timelines. A reliable, high-capacity grid ensures consistent power supply, and areas with recent upgrades or redundancies are particularly attractive. Some municipalities have long-range projects to ensure their power grid can meet the skyrocketing energy demands of datacenters.

These factors make power supply planning a crucial and complex component of AI supercluster development, often determining the feasibility and timeline of projects. The ability to secure adequate and reliable power is frequently a decisive factor in choosing locations for new AI superclusters.

Access to High-Capacity Power

A supercluster requires a massive amount of electricity, often measured in megawatts. Being close to a power plant, utility substation, or renewable energy source (e.g., hydroelectric, solar) can reduce transmission losses and is essential for reliable power.

For instance,

• Quebec, Canada, has ample hydroelectric power, which not only provides a renewable source but also offers favorable energy rates, attracting companies like Google and Microsoft.

• Norway’s hydroelectric power has also made it a highly attractive location for energy-intensive facilities.

Superclusters, particularly those hosting high-density GPU systems, can demand upwards of 100 MW. Grid stability and redundancy are key. A region with a reliable power grid and minimal outage history ensures that the supercluster can operate continuously without frequent interruptions.

To support such power consumption, planners often seek locations near robust power grids with excess capacity, as well as sites where new substations can be built if necessary.

For instance,

• Northern Virginia and Oregon are examples of areas with power grids capable of supporting these high demands.

• Singapore has invested heavily in ensuring its power grid stability, making it an attractive location for datacenters despite its relatively high energy costs.

Renewable Energy Access

Access to renewable energy is becoming a more key factor for companies looking to meet sustainability goals while also managing long-term operational costs. Hybrid energy strategies that combine renewable energy with conventional power can ensure reliability during periods of low renewable production.

For instance,

• In Denmark, Apple's Viborg datacenter combines renewable wind energy with grid power as a backup.

• In Ireland, several datacenters utilize a mix of wind and conventional power to maintain stability during fluctuations in renewable output, thereby ensuring continuous operations.

Renewable energy access not only offers a more stable pricing structure but also contributes to the reduction of the carbon footprint associated with superclusters.

For instance

• Iceland has abundant renewable geothermal and hydroelectric power.

• In Norway, the use of hydroelectric power and government incentives for renewable energy have led companies such as Facebook to establish datacenters.

Energy Costs and Long-Term Agreements

Energy costs can make or break the financial feasibility of a datacenter. Superclusters consume enormous amounts of electricity, which means even slight differences in energy pricing can translate into key cost variations over time.

• Subsidized Energy Rates: Proximity to power sources can sometimes provide access to lower energy rates, which is beneficial given the supercluster's high energy consumption. In addition, some regions offer subsidized energy rates or incentives to attract datacenter investments.

  • Texas offers competitive energy rates and incentives, which has led to rapid growth in datacenter development.

  • Washington State:

    • Incentives: Public Utility Districts (PUDs) in Washington, like Chelan County PUD and Grant County PUD, offer some of the lowest electricity rates in the US, thanks to abundant hydropower.

      They also offer incentives specifically for data centers, such as discounted rates for large loads and long-term contracts.  

• Long-Term Power Purchase Agreements (PPAs): PPAs with energy suppliers can lock in favorable rates for electricity over a multi-year period… typically 10 to 15 years. These agreements reduce exposure to market volatility and ensure a steady power supply, impacting the overall cost-efficiency of the site. For example

  • Google signed a 10-year PPA with AES for 125 MW of solar power for Google's data center in Santiago, also a 10-year PPA with Neoen for 125 MW of wind power for it’s datacenters in Finland.

  • Microsoft signed a 200 MW solar PPA with Engie in Texas, a 10-year PPA with Vattenfall for 280 MW of wind power for datacenters in Sweden, and 15-year PPA with Voltalia for 128 MW of solar power datacenters in France.

  • Facebook (Meta) signed a PPA with Apex Clean Energy for 100 MW of wind power for a datacenter in Nebraska, a PPA with Longroad Energy for 200 MW of wind power for datacenter in Los Lunas, New Mexico, and a long-term PPA with Ørsted for 140 MW of wind power for a datacenter in Luleå, Sweden.

These are just a few examples of the many PPAs these companies have in place. They often have a portfolio of PPAs with different energy suppliers, project sizes, and timeframes to ensure a reliable and cost-effective energy supply for their operations.

Summary of Power Considerations

The power supply is often the determining factor in the timeline for commissioning a datacenter. Ensuring early alignment with power suppliers, considering grid stability, and locking in power contracts are key actions to minimize delays. These aspects collectively contribute to the overall performance and reliability of the supercluster.

Securing power contracts is often the greatest bottleneck in the deployment of new megawatt-level datacenters. This process can take considerable time, and delays can significantly impact overall project timelines. Ensuring early negotiations and alignment with utilities is key to mitigating this potential roadblock.

Ensuring a stable and efficient power supply is key for a supercluster's success. Factors like grid capabilities, renewable energy access, and long-term agreements are all essential to achieve reliability, cost-effectiveness, and sustainability.

3. Cooling Requirements and Site Selection

Comparative Analysis of Cooling Techniques

Different advanced cooling techniques are suitable for specific climate conditions. Choosing the appropriate cooling method based on local climate can significantly improve efficiency and reduce costs.

For instance,

• In Singapore, the use of liquid cooling is more effective given the high ambient temperatures.

• In Iceland, air cooling is more efficient due to the cold climate.

• In California, evaporative cooling is preferred due to the dry air, which enhances the efficiency of the cooling process.

Large-Scale Cooling Solutions Leveraging Natural Resources

As GPU-based systems achieve greater and greater density, and generate greater heat, liquid cooling is becoming not an option but mandatory.

Leveraging natural resources for large-scale cooling solutions is a key factor when determining where to locate a supercluster. This not only reduces operational costs but also aligns with sustainability initiatives by minimizing the use of mechanical cooling systems. However, it is key to consider the potential environmental impacts of such solutions.

Large-scale use of natural resources, such as river or fjord water, may impact local ecosystems. Compliance with environmental regulations, such as EU's Water Framework Directive, and active ecosystem monitoring are essential to mitigate these impacts.

Examples:

• Google's Belgium datacenter includes a water treatment facility to ensure discharged water meets environmental standards.

• Hamina, Finland is notable for its use of seawater to cool Google's datacenter. This approach reduces reliance on energy-intensive mechanical cooling, utilizing nearby natural resources for efficient heat dissipation.

• British Columbia, Canada datacenters benefit from the cool air of nearby mountain ranges

• Dublin, Ireland, capitalizes on its temperate climate to naturally cool its datacenters.

• Norway utilizes fjord water to provide a constant source of cold water for datacenter cooling, as seen in facilities run by Green Mountain.

Implications for Site Selection

Cooling is one of the most key operational costs for superclusters. The amount of heat generated by thousands of GPUs requires advanced cooling systems, and the site's climate plays a role in cooling strategy:

• For instance, Cooling Considerations: Hamina, Finland leverages seawater for efficient datacenter cooling, as seen with Google's facility, while Oregon benefits from its cooler climate that allows for free air cooling. These examples show how natural climate and proximity to water sources can significantly impact cooling strategy.

• Liquid Immersion Cooling: Emerging technologies such as liquid immersion cooling are gaining traction, especially for GPU-heavy workloads. Immersion cooling involves submerging electronic components in a thermally conductive dielectric liquid, allowing for more effective heat dissipation.

  • Microsoft has tested this technology in its Washington state facilities to help manage extreme GPU workloads efficiently.

Summary of Cooling Requirements

Cooling is not just about mitigating heat; it plays a key role in optimizing overall system efficiency and reducing operational costs. Cooling strategies must align with local environmental factors, and leveraging natural resources where possible contributes to long-term sustainability.

Cooling efficiency is vital for supercluster operation. Leveraging natural resources, advanced cooling technologies, and appropriate site planning can significantly reduce costs and enhance sustainability.

4. Natural Cooling Resources and Water Consumption Considerations

Sustainability of Water Use in Drought-Prone Areas

Water consumption for cooling is a key concern, particularly in regions prone to drought. Selecting a site with sustainable water usage options and minimizing water dependency can be a deciding factor in site selection.

For instance,

• California datacenters are increasingly using recycled water for cooling to minimize their impact on local water supplies.

• Arizona has adopted similar practices, utilizing gray water for cooling to reduce dependence on fresh water sources.

• Datacenters in Australia have implemented closed-loop cooling systems to ensure minimal water wastage in a region frequently affected by drought.

Utilizing Natural Cooling Resources

Natural cooling resources, such as large bodies of water or cold air from high altitudes, can be leveraged to reduce the datacenter's cooling costs. By using water cooling systems, superclusters can efficiently dissipate heat generated by high-density GPU racks.

For instance

• The Hamina, Finland datacenter utilizes nearby seawater, illustrating how proximity to natural cooling resources can reduce operational costs.

• Ireland uses its naturally cool, moist climate to help with air cooling, reducing the reliance on mechanical cooling systems.

Air vs. Water Cooling Comparison: Depending on the geographic location, different natural cooling methods can be advantageous. For example, air cooling might be more feasible in arid, cooler climates, whereas water cooling could be beneficial near large natural water sources. Evaluating the pros and cons of each method helps in determining the most cost-effective and efficient solution.

• In Canada, particularly in British Columbia, datacenters make use of cool air from nearby mountain ranges, significantly reducing cooling-related energy consumption.

Environmental Impact: Access to natural cooling resources comes with environmental considerations. Using a lake or river for cooling requires adherence to local environmental regulations, including water discharge limits and maintaining water quality.

• Google’s datacenter in Belgium makes use of local river water, but also implements a comprehensive water treatment system to ensure that environmental standards are met before discharging used water.

Sustainable Water Usage Practices

Superclusters typically consume vast amounts of water for cooling, necessitating careful planning around water sources. Sites with access to sustainable water supplies, such as recycled water facilities or desalination plants, are preferred to minimize environmental impact. Emerging sustainable practices, such as using closed-loop water cooling systems that recycle water, help mitigate environmental concerns and align with regulatory requirements.

• California has stringent regulations regarding water use, which has led datacenters in the state to adopt recycled water for cooling to comply with conservation standards.

• In Singapore, some datacenters have adopted desalination and closed-loop systems to minimize their impact on local water resources.

• In Dublin, datacenters have taken advantage of proximity to the River Liffey while ensuring compliance with strict EU water usage and discharge standards.

Summary of Water Considerations

Water usage considerations, especially in regions prone to drought, impact both the feasibility and long-term sustainability of datacenter operations. Adopting sustainable water practices aligns with broader environmental goals and supports compliance with regulatory requirements.

Water usage is a key factor in site selection. By using sustainable practices and leveraging natural cooling resources, superclusters can effectively manage environmental impacts and ensure long-term operational feasibility.

5. Network Infrastructure and Connectivity Requirements

Proximity to Internet Exchange Points (IXPs)

If the supercluster spans multiple datacenters, the clusters must be connected to high-speed network infrastructure to handle massive data throughput and support low-latency operations essential for AI workloads.

Locating the supercluster near key IXPs ensures minimal latency and high bandwidth for data exchange. This proximity is particularly key for AI applications that require real-time data processing. For instance,

• Ashburn, Virginia, often known as 'Data Center Alley,' is a prime example of proximity to key Internet Exchange Points (IXPs) and a robust fiber-optic network.

• In Frankfurt, Germany, the presence of a key IXP has made the city a hub for datacenter operations, including those of Equinix and Digital Realty.

The presence of robust fiber-optic networks is key for maintaining high-speed data transfer between the supercluster and the rest of the internet. Sites with access to multiple fiber providers are preferred to ensure network redundancy.

• London has a well-developed fiber-optic network infrastructure, which supports numerous datacenters by providing resilient, high-speed connections.

Network infrastructure is a foundation of reliable datacenter operations. Effective strategies for redundancy, robust connectivity, and proximity to IXPs all contribute to maintaining the required performance for AI and high-density computing applications.

6. Geopolitical Considerations

Risks Associated with Geopolitical Instability

Geopolitical instability poses a risk to the long-term viability of datacenter investments. Potential risks include tariffs, restrictions on hardware imports, and political decisions that could impact operational stability.

For instance,

• In Hong Kong, datacenters face risks due to ongoing political tensions, which can lead to regulatory changes affecting data storage and processing.

• Similarly, United Kingdom datacenters have had to adapt to new trade and import restrictions resulting from Brexit, which has impacted the supply chain for necessary hardware components.

Geopolitical Stability and Economic Incentives

Political stability ensures the longevity of investments, while favorable economic policies can significantly impact the cost-effectiveness of the datacenter.

• Sweden offers political stability, robust infrastructure, and government incentives, making it an ideal location for datacenter investments.

• Singapore presents a stable geopolitical environment with strong regulatory frameworks, making it a favored destination for multinational corporations to host their datacenters.

• Ireland provides a favorable economic environment, including attractive tax policies and minimal bureaucratic obstacles for construction permits.

6. Municipal Environmental Impact & Sustainability

When selecting a location, evaluating the environmental impact and sustainability initiatives at the municipal level is key. Municipal policies regarding emissions, waste management, and energy use can influence the feasibility and attractiveness of a site.

For instance

• Amsterdam, Netherlands, has stringent environmental regulations to ensure that datacenters meet energy efficiency standards, including the use of renewable energy sources and efficient cooling systems.

• In Zurich, Switzerland, local governments encourage the reuse of waste heat from datacenters to supply district heating, which supports sustainability efforts while reducing environmental impact.

• In California, specific municipalities mandate water recycling in cooling systems to mitigate the environmental impact of water consumption.

6. Government Incentives and Energy Subsidies

Governments may offer various incentives, such as energy subsidies, tax breaks, and grants, to attract large-scale computing facilities to their regions. These incentives can significantly impact the economic feasibility of a project.

For instance,

• Energy Subsidies: Regions with subsidized energy rates can provide key cost savings over the datacenter's lifespan.

  • Sweden has established attractive energy subsidies to promote the use of renewable energy, which has helped draw datacenter investments from companies like Facebook.

  • Norway has attracted datacenters due to generous energy subsidies for renewable power usage, aligning with sustainability initiatives.

  • Quebec, Canada offers a program that provides a significant discount on electricity rates for data centers that meet certain energy efficiency criteria.

• Tax Breaks and Carbon Credits: Some countries provide tax credits for renewable energy use or carbon offset programs, aligning with the supercluster's sustainability initiatives.

  • Ireland offers favorable tax policies, attracting numerous datacenter projects

  • Denmark offers tax incentives for renewable energy-powered datacenters, which has helped attract Apple to build its datacenter in Viborg.

  • Oregon offers property tax exemptions for data centers located in enterprise zones. Oregon has seen significant growth in its data center sector, particularly in areas like Prineville, where Facebook has a major data center campus.

• Cutting Red Tape: Malaysian government is streamlining the permitting process for datacenters. Johor state of Malaysia has space, energy, water for cooling, and is near Singapore, a global communications hub.

  • Microsoft plans to invest $2.2 billion.

  • Oracle plans to invest $6.5 billion.

7. Land and Real Estate Cost

The cost, availability, and nature of land significantly influence the feasibility and design of a data center. As the data center footprint increases, careful consideration must be given to how land characteristics align with operational goals, financial constraints, and long-term strategic plans.

High-Density, High-Cost Locations

Though not a typical choice of location for AI model training given the scale required e.g power and cooling requirements. However, as AI applications ramp up across industry, superclusters for inference may be required in urban locations to be geographically closer to users.

For example,Singapore is a global business hub known for its advanced digital infrastructure but has a very limited land area. Real estate costs are among the highest in the world, forcing data centers to adopt a vertical architecture by constructing multi-story facilities. While this helps maximize the limited space, it also introduces unique challenges:

• Structural Engineering: Building multi-story data centers requires reinforced structures to support the heavy weight of servers and cooling equipment, increasing construction costs. Specialized engineering solutions, such as elevated floor designs for airflow management, are often necessary.

• Power and Cooling Distribution: Ensuring efficient power and cooling distribution across multiple floors adds complexity to the infrastructure. Each floor may need its own power distribution units (PDUs) and cooling systems, which can increase capital expenditures.

The limited space and dense urban setting may complicate the logistics of transporting and installing large equipment. Additionally, urban traffic and limited access routes can delay maintenance and emergency response times.

In high-cost areas, data center operators may employ modular design, constructing facilities in phases to spread out capital expenses. They may also explore leasing options for existing buildings and converting them into data centers to reduce upfront real estate costs.

Low-Density, Low-Cost Locations

Texas offers large, affordable tracts of land, which is advantageous for constructing sprawling, single-story data centers with high power density.

• Scalability: The availability of expansive land makes it easier to plan for future expansion. Data centers can be designed in a modular manner, allowing operators to add new server halls or wings as demand increases without significant redesigns or operational disruptions.

• Zoning and Land Use: In areas with more lenient zoning laws, data center operators can benefit from fewer restrictions on building height, equipment placement, and cooling unit noise. However, it's crucial to conduct thorough due diligence to ensure compliance with environmental regulations, especially concerning water usage for cooling.

While land costs are lower, other factors such as proximity to fiber optic networks and reliable power sources become crucial. In remote areas, the costs of establishing high-speed connectivity and ensuring a stable power supply can offset the savings from cheaper real estate. Additionally, regions prone to extreme weather conditions, like hurricanes or heat waves, necessitate investing in disaster-resistant infrastructure.

Industrial Zones and Reclaimed Spaces

In tech-centric regions like Silicon Valley, real estate is scarce and costly, so data center operators often repurpose industrial buildings or **warehouses to house their facilities. These reclaimed spaces provide some cost benefits, but operators must consider:

• Retrofitting Costs: Existing buildings may need extensive modifications to support the weight of server racks, implement proper cooling systems, and ensure robust power distribution. Retrofitting old structures can be nearly as expensive as new construction due to the need for structural reinforcements and upgrades to meet modern standards.

• Site Location: Industrial zones are often closer to city centers and business districts, reducing latency for services that require rapid data access. However, these zones may face restrictions on noise levels, emissions, and operational hours, requiring compliance with strict local regulations.

Urban vs. Rural Trade-offs

• Urban Data Centers: Typically closer to end-users, resulting in lower latency and faster data processing. However, they come with higher real estate costs, space constraints, and more stringent building codes. Urban data centers often need to be more energy-efficient due to limitations on power supply and cooling options.

• Rural Data Centers: Offer more space for large-scale operations and future expansion, as well as lower land and energy costs. However, they may face challenges in terms of connectivity, access to skilled labor, and potential increases in network latency due to their distance from major population centers.

In summary, land and real estate costs play a critical role in data center site selection and design. Operators must weigh the benefits of affordable, expansive rural land against the convenience and connectivity of urban locations. Understanding the interplay between land costs, utility access, regulatory requirements, and physical characteristics is crucial to ensuring a cost-effective, scalable, and reliable data center operation.

8. Workforce Availability

Access to a skilled workforce is another key factor in site selection. Datacenters require highly skilled labor for operations, maintenance, and ongoing development. Regions with a talent pool specializing in networking, AI, and data management are highly desirable for locating a supercluster.

For instance,

• Northern Virginia is home to a highly skilled workforce, partly due to its proximity to Washington D.C. and the presence of multiple technology firms and educational institutions.

• Bangalore, India, also provides a vast talent pool of IT professionals and engineers, making it a desirable location for datacenter facilities.

• Dublin, Ireland, benefits from its educational institutions and incentives for technology training programs, which contribute to a robust pipeline of skilled labor for datacenter operations.

9. Data Sovereignty, Privacy Laws, and Cross-Border Regulations

The location of a supercluster is influenced by data sovereignty and privacy laws, which dictate how data can be stored, processed, and transferred across borders.

Regulations like the GDPR (Europe) and CCPA (California) impose strict data handling requirements, which can impact site selection. A supercluster must be located in a jurisdiction that allows compliance with these regulations, particularly for AI applications involving personal data. For instance, France has strict data localization rules, prompting some companies to establish datacenters within the country to handle sensitive user information.

• Microsoft and Amazon Web Services (AWS) have established datacenters in Germany to comply with the European Union's GDPR, ensuring data sovereignty for European clients.

• Google has built datacenters in Switzerland to leverage the country's strict data privacy laws, appealing to customers with high data sensitivity requirements.

• Alibaba has set up datacenters in Malaysia to meet local data residency requirements and ensure compliance with regional privacy laws.

Conclusion

Selecting a site for a city-scale supercluster requires balancing political, geographical, environmental, infrastructural, economic, regulatory, and data sovereignty factors

Critical technical and environmental considerations include proximity to power and cooling resources, climate stability, and access to high-capacity grids and networks.

By evaluating these elements, planners can optimize locations for efficiency, sustainability, and operational success, while ensuring compliance and resilience.

In the next article, Article 20, “Conclusion”, our final article of this 20-part series, we'll synthesize key insights from our comprehensive exploration of AI superclusters, providing a holistic view of the infrastructure enabling deployment of massive compute for AI training.