If you’re asking your data engineers to demonstrate business value or find impactful projects to work on, you don’t have anything to offer them and they should just quit.
Finding impactful projects that show business value is the job of managers, PMs and leaders.
You’ll get value from engineers when you ask them to solve tough problems that directly impact the business. Building a dashboard or ETL job is not it.
You’ll get value from engineers when you give them sufficient business and technical context, but not limit them with too many fine-grained requirements.
Ok that’s pretty obvious, what’s less obvious is
You’ll get value from engineers when you place them on a team that can materially influence the business unit they are helping. A central data team (in a medium/large org) is a servant to many masters with the incentive to pump out code or dashboards or whatever and move to the next thing, not to drive the business forward. Assigning engineers directly to business teams, like marketing, sales, etc. will automatically make their work more impactful.
You’ll get value from engineers when you assign them a manager that deeply understands the business. A manager that is the best at eliminating business and organizational roadblocks. They don’t need to unblock technical issues, that’s why you have architects and principal/staff engineers. They need to set realistic goals and clear the path for projects to go out the door on time.
You’ll get value from engineers when they are part of the feedback loop. Lessons learned must disseminate throughout the org, down to every engineer, not just leaders, managers or PMs. Engineers will learn these lessons and iterate quickly.
Data engineers need to be allocated to where they can best serve the business.
Those who build and maintain infrastructure, tools, self-serve portals, etc. should ideally join the platform team. Data/analytics tooling and dev experience is not unique and doesn’t require its own team. Leverage the smart engineers that are already building dev tools for your software teams.
Those who build data products or solve business problems using data and analytics, should ideally join the specific business teams they are best suited to help. Imagine your sales department having their own data/analytics engineer that understands their needs, datasets and challenges and is dedicated to making them successful? Your sales leader will kiss you, their team could move so much quicker; btw this is not the same as having a SalesOps person.
If you’re an executive, realize the idea of a “data team” is rigid. It’s built on the same antiquated ideas that gave birth to the central IT team. For startups it’s less of an issue because engineers wear multiple hats and there is less hard delineation between functions. As the org grows, it’s important to resist centralizing the data function. The primary DRY part of it is infrastructure and tooling. Everything else is unique to specific business needs and shouldn’t be centralized.
If you’re a data engineering manager, consider splitting up the team and assigning them to teams that align better with business needs. Yes it may mean you eliminate your own position. It also means that you make your engineers much more impactful and find yourself in a bigger, also more impactful, leadership role.
If you’re a data engineer, pick a path (infra/devEx or business-aligned) and join the appropriate team. You will automatically start to work on impactful, valuable problems.
Data is a foundational building block of a modern business. From data stems better products, automation, prediction and a whole lots of goodness (and some badness too). Because data is so ingrained in everything a business need and does, centralizing the function of integrating data is not effective. Just like software engineers started as a cost-center and eventually were integrated into every business, do will data engineers.
This is the only future.