Sr. Server Hardware Systems Engineer (Starlink)

Employer description

SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today SpaceX is actively developing the technologies to make this possible, with the ultimate goal of enabling human life on Mars.

SR. SERVER HARDWARE SYSTEMS ENGINEER (STARLINK)

Starlink delivers high-speed, low-latency internet to millions of people around the world, and the network depends on a global fleet of servers running in data centers and points of presence (PoPs). The Starlink Site Reliability Engineering (SRE) team is hiring a Server Hardware Systems Engineer to own the health of this fleet as it scales rapidly. You will own the server hardware lifecycle from issue detection, to repair, through recovery. You will build the telemetry, alerting, and automation that let a small team operate a very large fleet, and coordinate with network operations, field technicians, hardware vendors, and data center partners to get failed hardware repaired and back into service quickly. You will also use fleet reliability data to shape server design, selection, and qualification for new sites across both commercial-off-the-shelf (COTS) and custom-designed server platforms. The right engineer for this job is one who is automation-first. Our goal is to scale operations for server hardware through excellent tooling, and automated process. While this role is primarily located in our offices (not in data center facilities), you are welcome and encouraged to spend time in the field with Starlink data center engineering teams

RESPONSIBILITES:

• Own the end-to-end hardware repair lifecycle for the Starlink server fleet and its components (CPUs, memory, SSDs, NICs, optics, and power supplies): triage alerts, isolate failures to the component level, and drive safe removal from production, repair, validation, and return to service
• Manage vendor support cases and coordinate on-site work with data center remote hands and field technicians, including parts delivery, site access, and return of failed parts for warranty replacement or failure analysis
• Build the software and automation behind that path: hardware telemetry and alerting (BMC and Redfish exporters, node-level metrics, log collection), removal of manual steps from diagnostics, production drain and return, alert silencing, and post-repair validation, and component-level tracking from detection to recovery
• Define, measure, and report fleet hardware metrics, including Mean Time to Respond, Repair, and Recover (p50 and p90), backlog trends, and component failure rates, and publish regular status updates to stakeholders.
• Write runbooks, methods of procedure (MOPs), and training material so NOC and field technicians can resolve common hardware incidents
• Validate new server hardware before it enters production, including burn-in, firmware, and configuration checks, and use fleet failure trends to drive server design, selection, qualification, and corrective action with vendors and hardware engineering

BASIC QUALIFICATIONS:

• Bachelor’s degree in computer science, information systems/IT, or an engineering discipline and 5+ years of professional experience in systems administration, site reliability engineering, or DevOps; OR 7+ years of professional experience in systems administration, site reliability engineering, or DevOps in lieu of a degree
• 3+ years of professional experience troubleshooting and repairing enterprise server hardware in a production environment
• 3+ years of professional experience administering Linux systems

PREFERRED SKILLS AND EXPERIENCE:

• 3+ years of experience operating a large server fleet in production data center environments
• Hands-on experience diagnosing and replacing server components such as CPUs, DIMMs, NVMe SSDs, NICs, transceivers, power supplies, and system boards
• Experience with out-of-band management and baseboard management controllers (e.g., HPE iLO, Dell iDRAC, OpenBMC) using Redfish and IPMI
• Proficiency with Linux hardware diagnostics, such as kernel and machine check logs, EDAC, ipmitool, smartctl, nvme-cli, ethtool, etc
• Software development experience in Python or Go, including building production tools, services, and automation
• Experience building monitoring and alerting with Prometheus, Grafana, and Alertmanager, including writing custom exporters
• Experience operating Kubernetes clusters, including node lifecycle tasks such as cordoning, draining, and replacing nodes
• Experience with server networking, including NICs, optics, cabling, and link-level troubleshooting
• Experience managing vendor support cases, RMAs, and warranty processes, and coordinating remote hands across multiple data centers and time zones
• Familiarity with firmware management, TPM enrollment, secure boot, UEFI, and server provisioning at scale
• Experience designing operational workflows in ticketing systems, and building dashboards for operational metrics such as MTTR and backlog size
• Experience with server hardware qualification, burn-in testing, or new product introduction
• Excellent written and verbal communication skills, with the ability to drive work to completion across teams, vendors, and time zones
• Ability to work in a fast-paced environment and manage multiple concurrent repairs across regions
• Experience writing scripts or software in at least one language, such as Python, Go, or Bash

ADDITIONAL REQUIREMENTS:

• Must be willing to work extended hours and weekends as needed to support hardware around the world.
• Must be willing to participate in an on-call rotation for hardware issues.
• Must be willing to travel domestically and internationally to data centers and other Starlink sites as needed, and must hold or be able to obtain a valid passport.
• Must be able to work in data center environments for extended periods, including standing, kneeling, and working in loud, variable-temperature conditions.
• Must be able to lift, push, and pull 50lbs.

COMPENSATION AND BENEFITS:

Pay Range:

Level 3: $200,000.00 - $280,000.00

Your actual level and base salary will be determined on a case-by-case basis and may vary based on the following considerations: job-related knowledge and skills, education, and experience.

Base salary is just one part of your total rewards package at SpaceX. You may also be eligible for long-term incentives, in the form of company stock or long-term cash awards, as well as potential discretionary bonuses and the ability to purchase additional stock at a discount through an Employee Stock Purchase Plan. You will also receive access to comprehensive medical, vision, and dental coverage, access to a 401(k) retirement plan, short and long-term disability insurance, life insurance, paid parental leave, and various other discounts and perks. You may also accrue 3 weeks of paid vacation and will be eligible for 10 or more paid holidays per year. Employees in Washington State accrue paid sick time in compliance with state and federal law. Company shuttles are offered to employees for roundtrip travel from select Seattle locations to the SpaceX Redmond office Monday to Friday.

ITAR REQUIREMENTS:

• To conform to U.S. Government export regulations, applicant must be a (i) U.S. citizen or national, (ii) U.S. lawful, permanent resident (aka green card holder), (iii) Refugee under 8 U.S.C. § 1157, or (iv) Asylee under 8 U.S.C. § 1158, or be eligible to obtain the required authorizations from the U.S. Department of State. Learn more about the ITAR here.  

SpaceX is an Equal Opportunity Employer; employment with SpaceX is governed on the basis of merit, competence and qualifications and will not be influenced in any manner by race, color, religion, gender, national origin/ethnicity, veteran status, disability status, age, sexual orientation, gender identity, marital status, mental or physical disability or any other legally protected status.

Applicants wishing to view a copy of SpaceX’s Affirmative Action Plan for veterans and individuals with disabilities, or applicants requiring reasonable accommodation to the application/interview process should reach out to [email protected].