Procurement details: Partnering with London Borough of Newham to Develop an Enterprise Cloud Data Platform on Microsoft Azure

• Summary of work:

  London Borough of Newham aims to lead in digital experiences for residents, staff and partners by utilising data and technology to enhance council services. To achieve this, the Council requires leveraging its Azure tenant and establishing a centralised, modern data platform. The project is fundamentally about transforming our relationship with data, creating quick and easy access to organisational information and improving our analytical reporting and transformation capabilities. The Council collects vast amounts of data relating to our residents and the services we provide. However, this data is currently held in multiple legacy systems across the organisation, with limited capability in our existing data infrastructure to access and deliver it at an organisational level. This creates several challenges for stakeholders needing this data to conduct business as usual, resulting in time-consuming, repetitive and inefficient manual processes. The new data platform must be at an enterprise level, providing the capability to bring together data from a large and complex legacy data estate in use at the Council. The key goals of the project will be: Pre-requisite configuration of Azure services within our existing Azure tenant as required for a public cloud platform. Although this is an optional requirement at this stage to be confirmed by the Council, the partner should have expertise in Azure infrastructure and be able to offer specialised knowledge and resources to enable the delivery of a high-quality data platform. Any specific configuration of Azure services needed will be confirmed at a later stage. Development and implementation of a Microsoft-based centralised cloud data analytical platform using Microsoft Fabric, Data Factory or a hybrid architecture with full compatibility with Power BI, our chosen BI solution. The proposed solution should also be future-proof and scalable to enable innovative technologies such as AI to be seamlessly embedded at a later stage. Data integration from source systems which will involve a combination of structured and unstructured data from our on premise systems (Oracle, SQL Server), SaaS applications (APIs) and external sources (flat files). As part of this engagement, we expect ingestion of data from these sources to build a minimum of six data products, which should include: • Finance Data • Social Care Data • Housing Data • Highways and Waste Services Data • Revenues and Benefits Data • Property Data As part of the development of these products, there will be a need to review our existing suite of reports driven by the legacy Data Warehouse to understand what data models are needed, either at an organisational or service level, to unify our analytical reporting and reduce reliance on the current Data Warehouse. A Master Data Management (MDM) data-matching exercise must also be undertaken during the transformation phase to ensure that data from disparate systems is reviewed and data quality issues are resolved to ensure that only the most current and accurate data is displayed. This should allow the Council to build and understand a single view of residents, households, and properties and identify any additional data sources required to create a complete view. The next goal is to configure Azure Purview to classify, catalogue, assign responsibilities, and track the lineage of data assets. Throughout the development of the data platform, there must be strict adherence to organisational data governance policies to ensure data is managed effectively, securely and in compliance with all relevant regulations. This includes conducting a Data Protection Impact Assessment (DPIA) and other security assessments for data processing activities. We also want to enable notebooks in Azure to create real-time insights into our resident data, understand trends and predict issues before they occur. Please note there is an existing small data team with whom the partner is expected to collaborate to develop their in-house capabilities and capacity to take ownership of both development and support tasks. The final requirement is to manage the new data platform for the first year, after which it will be handed over to an internal managed service provider in late 2025. Please note that there may be a need to migrate this platform to a new Azure tenant, which will be confirmed at a later stage.

• Where the supplied staff will work:

  No specific location (for example they can work remotely)

• Who the organisation using the products or services is:

  London Borough of Newham

• Why the work is being done:

  London Borough of Newham is in the early stages of developing a data strategy as part of its Data Modernisation programme. An enterprise level cloud-based data platform is a key component of this wider strategy. The Council is data rich and aims to provide a self-service approach for its stakeholders to forecast business needs, understand performance, identify issues and set priorities. Better, well-managed, trusted and easy-to-use data will enable us to provide the best possible services to those who rely on us, save money, help our residents, provide more support and better target our resources. Currently, our data infrastructure relies on an on premise data warehouse running SQL Server 2016, which serves as our core analytics system. This warehouse consolidates data from over 62 different source systems, including the matching of people, properties and businesses. While this data is accessible to staff via enquiry screens, the underlying technology is outdated and lacks the scalability and flexibility needed to handle the increasing volume and variety of data generated by the Council. Additionally, there are several on premise data marts designed for single-source data storage, supported by legacy SSIS. These data marts are challenging to maintain and often suffer from degradation, leading to operational inefficiencies and unreliable data outputs. Consequently, maintaining and upgrading these older systems presents significant challenges and limits our ability to integrate new data sources and adopt modern analytics tools effectively. The Council aims to transition to a modern cloud architecture that capitalises on advanced features for data processing and analytics. This move will provide flexible structures for organising and interpreting data that can adapt to changes in source systems, suitable connectors facilitate the integration of data from different sources, optimise data storage and enhance system robustness by reducing our on premise server footprint. By adopting this cloud-based approach, the Council wants to remain agile and responsive to evolving organisational needs and technological advancements. Additionally, this transition should help us address current issues with duplicate records, which vary in consistency and accuracy often leading to potentially incorrect and unreliable information that requires manual intervention. The shift to the cloud will also strengthen our security measures, improve our ability to protect sensitive data, and ensure compliance with relevant regulations. Enhancing our approach to data will lead to better resident-facing services, support more informed decision-making for both policy and operational matters and deliver insights that add value and drive efficiencies. The Council aims to improve its ability to use and reuse data effectively, particularly through initiatives like “Tell Us Once” services for our residents and communities and strengthen our business intelligence capabilities at an organisational level to drive the necessary transformation. To achieve these goals, we aim to have a partner in place by January 2025 to begin mobilising and to have the core enterprise data platform configured by April 2025. Full integration of data sources is expected to be completed over the 2025/2026 financial year.

• The business problem:

  Newham Council is collecting increasing volumes of data daily but our current architecture is outdated and lacks the scalability and flexibility needed to manage this growing volume and variety of information effectively. As our business environment rapidly changes, our legacy system is unable to adapt to shifts in operations, resulting in highly manual, time-consuming data analytics processes with limited AI and ML capabilities. Our stakeholders lack a clear view of the data we hold and there is low confidence in the quality and reliability of the available information. Our aim is to address these data silos and eliminate inefficiencies in how we manage and utilise data. We seek to transition to a modern cloud data platform that reduces maintenance costs, provides a flexible architecture to meet the Council's future needs and enhances productivity by allowing us to focus on innovation and growth. This project fundamentally seeks to transform how we work with data across all levels of the organisation. We intend to create a centralised, cloud-based data repository that offers quick and secure access to well-managed data across all council services, while maintaining top-tier security measures that are continuously updated to address potential threats proactively. We are looking for a solution that addresses data quality issues to detect errors or inconsistencies at the point of entry and help us correct problems with the source data, reducing our reliance on time-consuming manual corrections. Additionally, we aim to improve repetitive tasks related to data extraction and transformation from legacy data marts, which currently cause inefficiencies and errors. The new data platform should incorporate modern technologies and tools, ranging from advanced analytics using Power BI to AI solutions and predictive algorithms. We require an organised approach to governing, protecting and managing data effectively, resulting in a more efficient, scalable and reliable data management system that enhances analytical capabilities, supports informed decision-making, and integrates seamlessly with various technologies while allowing data sharing with key partners. Building the technical capability of our internal data team to effectively manage and utilise the Azure platform is crucial. Currently, our IT and data infrastructure are managed as a shared service between Newham and another council. However, both boroughs are in the process of separating their ICT departments and disaggregating the shared infrastructure. This creates additional challenges, requiring us to navigate the complexities and uncertainties around setting up Azure services. You will be expected to work closely with an interim Managed Service Provider to ensure that the correct services are configured within the Azure tenant to support the development of the new data platform.

• User type:

  Data Analysts

• Definition:

  Data Analysts across the organisation need to access data from our multiple legacy systems and various sources and ensure it is in a usable format. They require instant and continuous access to this data, in its most up-to-date availability to produce analytical reports for their various service requirements. Access to data from systems managed by other services will allow analysts to produce detailed reports and visualisation on relatable data to provide a clear and full picture to the effectiveness of the services we provide to our citizens. Data analysts require access to good quality data to ensure that their focus can be on delivering insights into our data as opposed to time-consuming and manual transformation of the data to prepare it for analytical use.

• User type:

  Performance leads

• Definition:

  View live dashboards, which are fed by constant streams of data from our various systems, to gain insights to council performance in relation to defined metrics and performance indicators. The ability to filter dashboards to view historic data and highlight potential emerging trends in our performance data will allow performance leads to make data-driven decisions and address potential issues before they arise.

• User type:

  Senior leadership

• Definition:

  Senior leadership must be able to interface with detailed and focused reporting fed by up-to-date, good quality data from the proposed solution to allow them to gain valuable insight into our data, through the identification or historical trends and potential future trends through predictive analytics. This will enable Senior leadership to make data-driven decisions.

• User type:

  Chief officers

• Definition:

  Chief officers must be able to interface with detailed and focused reporting fed by up-to-date, good quality data from the proposed solution to allow them to gain valuable insight into our data, through the identification or historical trends and potential future trends through predictive analytics. This will enable Chief officers to make data-driven decisions.

• User type:

  Council officers

• Definition:

  View live dashboards, which are fed by constant streams of data from our various systems, to obtain a single view of our citizens, combining data from multiple sources to visualise all data we hold on a citizen. This will allow for efficiencies in multiple processes, through the removal of manual data matching and sourcing of relatable data from other services and teams. Confidence in the data platform’s ability to present real-time data ensures that any decision made with citizen data is done so well-informed. Council officers should also have access to relevant data sources, through managed-access levels, to support a self-service first approach to data analytics within the council.

• User type:

  Elected members

• Definition:

  Elected members must be able to interface with detailed and focused reporting fed by up-to-date, good quality data from the proposed solution to allow them to gain valuable insight into our data, through the identification or historical trends and potential future trends through predictive analytics. Having access to these interactive dashboards will allow elected members clear and concise access to relevant data to support decision making.

• User type:

  Azure administrators

• Definition:

  Azure administrators must be involved in the development of this solution to understand the resources enabled and their configuration. Defined roles to grant and revoke access to specific datasets must be granted to these users to allow for management of the solution beyond the end of the project.

• Any pre-market engagement done:

  In February 2024, we conducted pre-market engagements with suppliers recommended by Microsoft to gain insights into how other local authorities have modernised their data architecture and analytical reporting.

• Which phase the project is in:

  Not started

• Existing team:

  There is currently a data team consisting of four members: the Head of Data, two Data Analysts and one Data Engineer. Additionally, there are BI and data analysts across various service areas within the Council. At present, our IT and data infrastructure are managed as a shared service between Newham and another council. However, both boroughs are in the process of separating their ICT departments and disaggregating shared infrastructure. In this context, you will be expected to work closely with an interim Managed Service Provider to ensure that the appropriate services are configured within the Azure tenant to support the development of the new data platform.

• Address where the work will be done:

  Much of the work can be delivered remotely. Where there is a requirement to visit the Local Authority office this will be at Newham Dockside, 1000 Dockside Rd, London E16 2QU.

• Working arrangements:

  The supplier staff will keep Newham Council Data team working hours in order to maximise collaboration, efficiencies and knowledge transfer. Remote working, communicating with Newham Council staff via Microsoft teams meetings, calls and messages. No scope for expenses to be provided as part of this project.

• Security and vetting requirements:

  Baseline Personnel Security Standard (BPSS)

• Latest start date:

  6 January 2025

• Contract length:

  2 years 0 months 0 days

• Optional extension:

  1 years 0 months 0 days

• Indicative maximum:

  £240000

• Indicative minimum:

  The contract value is not specified by the buyer

• Contracted out service or supply of resource?:

  Contracted out service: the off-payroll rules do not apply

• How many suppliers to evaluate:

  4

• Technical Competence:

  40%

• Cultural fit:

  10%

• Social values:

  10%

• Price:

  40%

• Essential skills and experience:

  18%

• Nice-to-have skills and experience:

  2%

• Technical questions:

  80%

• Description:

  Can you provide an example of how you have delivered an enterprise data platform and cloud infrastructure services as part of transformation programmes for complex, multi-sector local authorities?

• Weighting:

  10%

• Description:

  Can you describe your expertise in Microsoft Azure, specifically in utilising Azure’s data services and AI capabilities? Please include details on relevant certifications in analytics, digital and app development, and infrastructure

• Weighting:

  10%

• Description:

  Can you describe your expertise in deploying a Microsoft Fabric or Hybrid architecture for complex, multi-sector local authorities?

• Weighting:

  10%

• Description:

  Can you provide examples of how you have ensured data governance and compliance with GDPR and other relevant regulations when handling sensitive local authority data?

• Weighting:

  10%

• Description:

  Can you give an example of how you have worked with council applications to integrate fragmented data from various sources, creating a comprehensive view of residents and properties?

• Weighting:

  10%

• Description:

  Can you describe how you have integrated data from multiple sources such as housing, social services, and revenues and benefits, to provide a unified dataset for analysis and decision-making?

• Weighting:

  10%

• Description:

  Can you give an example of how you have tackled issues around data quality when the same information is held across disparate systems?

• Weighting:

  10%

• Description:

  Can you describe your experience of working effectively within complex, multi-sector local authorities, engaging stakeholders and overcoming resistance to data-sharing and breaking down data silos across services?

• Weighting:

  10%

• Description:

  Can you describe your experience in supporting internal staff to build in-house capabilities and ensure effective knowledge management and transfer, including training, coaching, and mentoring as needed?

• Weighting:

  5%

• Description:

  Can you give an example of your previous work with local authorities on using data to solve current challenges faced by councils?

• Weighting:

  5%

• Description:

  Can you give an example of your previous work with local authorities where you have implemented a data platform that enables AI and ML

• Weighting:

  5%

• Description:

  Can you give an example of how you have helped an organisation develop and implement a data strategy?

• Weighting:

  5%

• Description:

  Can you provide examples of how you have supported or enabled organisations to develop a single view of customers or households, or implemented platforms that facilitate automated "Tell Us Once" processes?

• Weighting:

  50%

• Description:

  Can you describe your experience following defined development practices throughout the stages of development, user testing and production environments?

• Weighting:

  50%

• Question:

  Can you provide a brief outline of the technology stack you propose for our data platform solution? Please cover all stages, including data extraction, transformation, loading (ETL), storage and analysis/reporting.

• Weighting:

  10%

• Question:

  How will you ensure that the proposed solution meets our specific requirements, including functional and non-functional aspects such as scalability, performance and compliance?

• Weighting:

  10%

• Question:

  Can you describe your approach and methodology for developing the proposed solution? Will you employ an agile development process and if so, how will stakeholders be engaged throughout the project lifecycle?

• Weighting:

  10%

• Question:

  What measures will you implement to ensure data is handled securely, particularly in compliance with GDPR and other relevant regulations?

• Weighting:

  10%

• Question:

  How would you define or adapt a common data model for each service area, such as supporting families, to align with our specific use cases?

• Weighting:

  10%

• Question:

  What steps will you take to address data quality issues, standardise reporting processes, and ensure consistency and accuracy? How could a lakehouse architecture be applied in our context?

• Weighting:

  10%

• Question:

  How will you ensure the platform’s security against cyber-attacks and restrict access to the platform from outside the Newham environment?

• Weighting:

  10%

• Question:

  How will you identify and manage risks and dependencies associated with implementing this solution? Can you provide examples of how you have worked with internal teams to resolve similar issues in previous projects?

• Weighting:

  10%

• Question:

  How do you plan to ensure the solution is future-proof and scalable, able to accommodate new data sources and evolving requirements?

• Weighting:

  5%

• Question:

  What expertise do you have in Microsoft Azure, and how would you leverage Azure’s data services to support our platform requirements? Please provide references or case studies from other public sector organisations where you have successfully delivered similar solutions.

• Weighting:

  5%

• Question:

  How will you support our internal teams in building in-house capabilities for managing the data platform post-implementation? Please describe your approach to training, knowledge transfer, and ongoing support.

• Weighting:

  5%

• Question:

  How would you reduce the complexity of our current on-premise data warehouse, particularly in relation to data matching and quality, during the migration to the cloud? Please outline your strategy for simplifying data architecture, optimising data workflows, and consolidating disparate data sources.

• Weighting:

  5%

• Question:

  Please present how will you ensure effective teamwork with our organisation. Please highlight your team structure and their expertise including approach to being transparent and collaborative in decision-making, fostering a no-blame culture and encouraging learning from mistakes. Additionally, how will you share knowledge and experience with other team members to support overall project success? (This question will be evaluated via presentation, further details will be provided in ITT)

• Weighting:

  65%

• Question:

  Please present how will you work with users who have lower technical understanding and address resistance to data-sharing from service-level data silos. Please detail your strategies for ensuring stakeholder buy-in across services and facilitating access to service data. (This question will be evaluated via presentation, further details will be provided in ITT)

• Weighting:

  35%

• Question:

  Theme 2: Tackling Economic Inequality Policy Outcome: Create New Businesses, New Jobs and New Skills MAC 2.2: How will you create employment and training opportunities particularly for those who face barriers to employment and/or who are located in deprived areas.

• Weighting:

  25%

• Question:

  Theme 2: Tackling Economic Inequality Policy Outcome: Create New Businesses, New Jobs and New Skills MAC 2.3: How will you support educational attainment relevant to the contract, including training schemes that address skills gaps and result in recognised qualifications.

• Weighting:

  25%

• Question:

  Theme 3: Fighting Climate Change Policy Outcome: Effective Stewardship of the Environment MAC 4.1 Please describe how you will deliver additional environmental benefits in the performance of the contract including working towards net zero greenhouse gas emissions.

• Weighting:

  25%

• Question:

  Theme 3: Fighting Climate Change Policy Outcome: Effective Stewardship of the Environment MAC 4.2 Please detail how you will influence staff, suppliers, customers and communities through the delivery of the contract to support environmental protection and improvement

• Weighting:

  25%

• Pricing model:

  Capped time and materials

• Additional assessment methods:

  Presentation

  Case study

  Work history

• Level:

  Not met

• Score:

  0

• Description:

  The response does not meet any of the requirements or no response has been provided. An unacceptable and / or non-compliant response with serious reservations, demonstrating no understanding of the requirement.

• Level:

  Partially met

• Score:

  1

• Description:

  The response has met some, but not all elements of the requirement, which poses risk that the proposal will not meet the deliverables required. The response does not demonstrate a full understanding of the requirement posing major concerns.

• Level:

  Met

• Score:

  2

• Description:

  The response is acceptable and meets all the basic requirements. However, the response is not sufficiently detailed to minimise risk and / or the proposed approach may require additional support (in addition to that outlined in the Statement of Requirements) from the Contracting Authority to meet its deliverables.

• Level:

  Exceeded

• Score:

  3

• Description:

  The response exceeds requirements, providing detail that minimises risks to delivery. The response is comprehensive and unambiguous, demonstrating a thorough understanding of the requirements and providing details of how the requirement will be met in full without additional support from the Contracting Authority, other than that outlined within the Statement of Requirements.

• Publication of stage 1:

  16/09/2024

• Clarification period closes:

  20/09/2024 16:00

• Deadline for suppliers to submit their stage 1 responses:

  30/09/2024 16:00

Return to Partnering with London Borough of Newham to Develop an Enterprise Cloud Data Platform on Microsoft Azure

Help

You can contact us by email, phone or using the enquiry form .

Email:

info@crowncommercial.gov.uk

Telephone:

0345 410 2222

CCS customer services team is available Monday to Friday, 9am to 5pm.