Edge Inference & Distributed Micro-Datacenters

Every other archetype in Part 1 chases power. Pre-training chases the cheapest stranded megawatt in the coldest climate, batch inference chases curtailable off-peak load, and even online inference will happily sit in a regional hyperscale campus an hour from the nearest user. Edge inference inverts the siting hierarchy. Its binding constraint can be a measured latency ceiling, a local data boundary, WAN-loss continuity or the cost of transporting inputs. Proximity beats cost, and that produces a different building: small, distributed, power-starved, thermally constrained and run with no one in the room. Done for the right reason, it serves workloads a region cannot serve at any price. Done for a latency budget that was never physical, it is the most expensive way to do something a region does better and cheaper.

We define the edge as a tiered topology: on-prem appliance, telco/MEC node, Tier-2 metro colo and CDN-adjacent grid. The tiers trade latency, cost and operational reach against each other, so picking the wrong tier is its own mistake. Everything here is anchored to the latency budget and the 30/50/100 ms perceptibility thresholds, as the edge archetype was in Chapter 1.1. From there we trace what the inversion forces: tight power, thermal and form-factor limits, lights-out fleet operations with zero-touch provisioning, and an economics question, edge versus a centralized region, whose answer is "centralized" far more often than edge enthusiasts admit.

"The edge" is the most abused word in this guide. It is a gradient from the user's premises back toward the centralized region. Each step outward trades lower latency and better data gravity for less power, worse economies of scale and harder operations. Four tiers cover almost every real deployment, and each is a different design problem.

On-prem appliance. A GPU box or sealed micro-rack inside the user's own building: a factory floor, a hospital, a retail back room, a trading desk, a forward military node. Latency is essentially zero because there is no WAN hop, and data never leaves the premises (the data-sovereignty and air-gap case). Power is whatever the building's electrical service can spare, cooling is ambient air or a sealed loop, and there is no IT staff. This is the tier for hard real-time control and for data that legally or contractually cannot cross a network.

Telco / MEC node. Compute inside the mobile operator's network, at the cell-site aggregation point, the central office or a regional breakout, under the ETSI Multi-access Edge Computing (MEC) framework. This is the tier 5G URLLC was built to feed. Access-edge placement can shorten the path, but the service budget must also cover the device's radio and access delay, transport, queueing and application processing. ETSI's January 2017 discussion gives a general 10 ms network-latency target that assumes negligible application processing; test the deployed device-to-MEC-server path at the percentile you need. Telco real estate is space-, power- and cooling-constrained by design (these rooms were sized for switching gear, not GPUs), and the operator owns the landlord relationship. AI-and-RAN offers a shared-hardware route: radio processing and AI workloads use the same accelerated platform. NVIDIA's 16 March 2026 announcement describes T-Mobile pilots with Nokia anyRAN, RTX PRO 4500 Blackwell for constrained cell sites and RTX PRO 6000 Blackwell for switching offices; Nokia reported broader operator lab and field trials on 16 September 2026. Before underwriting that shared capacity, compare the offered site's power and cooling limits, how GPU cost is allocated, and how much inference capacity stays reserved when radio demand peaks.

Tier-2 metro colo. A conventional colocation hall in a second-tier metro: not Ashburn or Santa Clara, but one of the dozens of mid-size cities that have users and no cloud region. It is the workhorse for latency-sensitive inference at meaningful scale. It has real power (hundreds of kW to a few MW), real cooling, operations staff within driving distance, and 10–30 ms reach to a metro's users. It is "edge" only relative to the hyperscale region; physically it is a normal data center placed where the users are.

CDN-adjacent grid. Inference injected into an existing content-delivery footprint of hundreds or thousands of points of presence (PoPs). The CDN tier inherits a distribution network and routing layer, while model fit depends on the GPU nodes and power at the serving location. Cloudflare reports that most of the Internet-connected population is within 20 ms of its network and 95% within 50 ms (network page, checked 5 October 2026). Budget the measured round trip to the GPU instance that actually serves the model, including routing and capacity placement. Cloudflare's catalog includes Llama 3.3 70B and gpt-oss-120b (checked 5 October 2026). Akamai's March 2026 AI Grid combines its 4,400-plus edge locations for caching, routing and serverless functions with dedicated RTX PRO 6000 Blackwell GPU clusters. Screen the actual model-serving location and capacity, and remember that the operator rents someone else's footprint, with someone else's power and cooling limits.

The latency budget governs the edge, so it is worth knowing what is in it and where the thresholds come from. End-to-end user-perceived latency is a sum of four terms: network round trip (dominated by physical distance), access-network overhead (the last mile: 5G, Wi-Fi, fixed broadband), queuing and processing at each hop, and the inference time itself (time-to-first-token plus enough decode to be useful). Edge placement reduces propagation and can remove intermediate-hop queueing and processing; on-prem serving can also avoid the WAN access path. Moving an unchanged model onto an equivalent accelerator does not reduce its computation time. If model execution alone exceeds the total latency budget, placement cannot meet the SLO; change the model or serving hardware, as described in Chapter 1.3.

The physical floor is worth memorizing. Light in fiber travels at roughly two-thirds of c, about 0.80 ms per 100 miles each way, so a user 600 fiber-route miles from the nearest region pays ~10 ms round trip on distance alone, before a packet is queued or a token generated. That number makes or breaks the edge case. Three practitioner bands recur, each mapping to a class of experience. They are design bands, not one measured universal: before committing to any of them, state the endpoints you are budgeting between (motion-to-photon, network round trip, inference, actuation) and whether you mean a percentile or a deadline.

• ~100 ms: the ceiling for an interaction to feel "instant" and for total response to preserve natural conversational flow. Above it, the system feels laggy but usable. This is the loosest threshold, and most regions already meet it for nearby users.
• ~50 ms: the working budget for genuinely interactive experiences such as AR/VR overlays, live translation, responsive agentic loops and cloud-rendered interfaces. A 50 ms p99 is the SLO most often cited as the line where edge placement starts to justify its cost, because little room remains once access-network and inference time are subtracted.
• ~30 ms and below: the band where a missed deadline has a physical consequence, which is what "hard real-time" actually names (a deadline and its consequence, not this number). Examples are industrial control, autonomous-vehicle perception assist, haptic tele-operation and safety interlocks. Here the compute is almost always on-prem or at the access edge. Distance alone does not forbid a regional round trip (applying Chapter 3.6's ~5 µs per route-km and 1.3–1.5× route factor, propagation alone leaves roughly 2,000 km of straight-line reach inside a 30 ms round trip). The access-network and inference terms consume most of the budget before propagation is even charged.

The consequence chain is direct. The threshold you commit to sets the maximum distance to the user, which sets the minimum number of sites, which sets the whole economics. A 100 ms budget might be served from a handful of regions plus a CDN front-end. A 30 ms budget for a national user base can force dozens to hundreds of micro-sites, and that site count, not the GPU bill, dominates edge TCO.

The inversion that defines the edge has a corollary: the places closest to users are the worst places to put power and heat. A hyperscale campus is sited because it has cheap, abundant megawatts and room for a cooling plant. A cell-site cabinet, a retail back room, a CDN PoP or a Tier-2 colo suite has neither. The edge is therefore power-bound at the micro scale, just as the industry is power-bound at the macro scale, but the binding number is kilowatts, not megawatts, and it is set by a building someone else designed for something else.

Power sets the model, not the other way around. In a centralized build you pick the model and provision power to match. At the edge the host site fixes the available power, and that decides which models can run there at all. RISE's networked edge data-center class is 5–50 kW; narrower site values are the guide's own screening assumptions, not a universal edge limit. A model that needs a complete NVL72 rack cannot live at a micro-site. NVIDIA's March 2026 GB300 NVL72 reference architecture requires up to 142 kW per rack, and NVIDIA's DSX reference design plans Vera Rubin NVL72 (VR200) cabinets at 330 kW TDP, the facility basis for power and cooling. OEM datasheets rate operating profiles lower: PEGATRON's RA4803-72N3 lists 188 kW at Max Q and 228 kW at Max P. Below that line, model fit is set by the qualified server's GPU memory, precision, context length, concurrency and latency target. Distilled models and larger quantized or sparse ones can fit: Cloudflare introduced Llama 3.1 70B serving in September 2024, and OpenAI's August 2025 gpt-oss-120b runs on a single 80 GB GPU. These are examples of fit, not a parameter ceiling. A metro colo, at hundreds of kilowatts to a few megawatts, can provision the full rack. In the power-constrained tiers (on-prem appliance, telco cabinet, CDN PoP), going edge means choosing a model that fits the qualified server and service budget. A workload that requires a complete NVL72 rack fails the micro-site power screen. → quantization and model sizing for serving: Chapter 1.3.

Thermal design is ambient-limited. The cooling-cliff logic of Chapter 5.1 still applies: facility-water availability selects the heat-rejection topology, not whether liquid can reach the chip. The dominant answer is sealed or closed-loop air, or self-contained liquid that rejects to ambient, in a modular cabinet that needs no plumbing to the building. That keeps the form factor deployable but caps the heat you can reject, which loops back to the power ceiling. Form factor is the third constraint: micro and modular facilities can package capacity for repeatable deployment, while each host site still needs the required electrical, network and heat-rejection interfaces.

A centralized region is one operations problem with staff on site. An edge deployment is the opposite: hundreds of sites, none staffed, scattered across a geography. This decision quietly kills more edge projects than latency or power, because the edge's cost is operational, not capital. A fleet you cannot run without sending a technician to each site loses money on every truck roll. The model works only if a site can be deployed, configured, monitored, updated and recovered with no one in the room.

Zero-touch provisioning (ZTP) is the enabling discipline. Hardware ships to a site, is plugged in and powered on, then enrolls itself, pulls its image and configuration from a central control plane, joins the fleet and starts serving, with no local expertise and no manual steps. Scale Computing's November 2022 data sheet claims a 90% or greater reduction in node and cluster installation time through central preconfiguration and automatic initialization after connection and power-on. Site preparation, power and network connections remain separate deployment work. Everything downstream of provisioning must also be remote: over-the-air model and firmware updates, remote attestation and secure boot (the physical perimeter at an unmanned site is weak, so the root of trust must be in silicon; see Chapter 11.4, the canonical home for hardware root of trust and firmware security), telemetry-driven health monitoring, and automated remediation that can quarantine, reimage or fail a site over without dispatch.

Getting this wrong does not give you a worse SLA. It gives you a different business. Without ZTP and remote remediation, every firmware bug, wedged node and certificate rotation becomes a truck roll, and operating cost scales with site count instead of being amortized across it. The automation platform is reversible, since you can re-tool. The decision to operate lights-out is irreversible at scale: you cannot retrofit hands-on operations onto a hundred unstaffed sites without rebuilding the cost model. → fleet reliability and goodput at the edge connect to Chapter 1.3 on inference operations.

Against a real workload, does the edge beat a centralized region with a CDN front-end? More often than its advocates concede, no. The edge trades away the three things that make centralized inference cheap, economies of scale, high utilization and cheap power, in exchange for local capabilities the region cannot supply within the workload's requirements: latency, data-boundary compliance, WAN-loss continuity or lower input-transport cost. Compare only placements that satisfy those requirements. If you do not need those capabilities badly enough, every other line of the comparison favors the region.

Utilization is the edge's structural weakness. A region pools demand from a continent and runs its fleet hot. An edge site serves only the users in its latency radius, so its demand is thinner, spikier and harder to fill. In an illustrative case, a regional fleet sells 80–90% of its available GPU-hours and a metro edge site sells 30–40%. With the same fixed cost per available GPU-hour, the edge site spreads that cost across far fewer sales. These are scenario assumptions, not measured fleet averages. A single metro's latency-sensitive demand must sell enough billable GPU-hours, at a price above variable cost, to cover the edge site's fixed operating cash and debt service. Use Chapter 1.8's matched cash-flow case to compare the capacity commitment and Chapter 2.5's DSCR case to test financing. Power is the edge's second penalty to check against real tariffs. Metro power typically costs about 1.5–2x the cheapest US industrial rates (EIA, July 2026). Virginia industrial power was 10.07¢/kWh against 7.07¢ in Texas and 6.08¢ in Louisiana, about 1.4–1.7x. Only California- or New Jersey-type markets (25.11¢ and 15.63¢) reach 2.5x or more. These are state sector averages, so price your own sites on their own tariffs.

The comparison has a clear shape. Centralized + CDN wins when the latency budget is soft (≥100 ms), demand is too thin to fill edge sites, the model is large, or the workload is batch-shaped, which describes most inference today. Edge earns its fleet premium when a region cannot meet a required latency, data-boundary or WAN-autonomy constraint, or when avoided input transport makes local processing cheaper overall, and the selected site's model, power, thermal and availability limits meet the workload. The mistake is treating the edge as a default modernization step rather than a targeted answer to a physical constraint. When in doubt, the region is the cheaper bet, and you can always push selected workloads outward later, which is itself the reversible move.

The same mis-scopes recur, each from inverting the siting hierarchy when the inversion was not earned:

• Edge for a preference budget. Distributing inference to shave latency the SLA never required. The region already cleared 100 ms for nearby users; the edge buys 40 ms of headroom nobody asked for, at the price of thin utilization and a fleet to operate. Compare a faster centralized accelerator and any WAN-autonomy or transport-cost benefit before paying for distribution.
• Edge with a model that does not fit. Committing to edge serving, then discovering the product needs the frontier model that an NVL72 hosts and a site-specific edge cabinet within the 5–50 kW networked edge DC class cannot. The power limit is the first line of the spec, not the last: decide the model and the site together.
• Edge without lights-out operations. Standing up a hundred sites with no ZTP, no remote remediation and no in-silicon root of trust, so every firmware bug becomes a truck roll and opex scales with site count. The automation must exist before the fleet does.
• Per-site redundancy instead of geo-redundancy. Trying to make each unmanned cabinet bulletproof with 2N power it has no room for, instead of placing an extra site in the coverage gap. Test the extra site's reach, state and spare capacity against the same failure the local 2N proposal addresses, and spend where the required service survives.