如欲瀏覽中文版本，請按此。
An article from the World Economic Forum points out the fact that we now live in a world wherein a day 500 million tweets and 294 billion emails are sent. For every connected car, 4 terabytes of data are generated. 65 billion messages are sent on WhatsApp, and 5 billion online searches are conducted online every day.
According to the Visual Capitalist, by the year 2025, the equivalent of 212,765,957 DVDs in data will be generated in a span of 24 hours. The pandemic of 2020 further boosted the amount of data being generated and consumed every day.
This trend has also influenced an economy like Hong Kong, which has always been a digital powerhouse, due to its technical capabilities and culture of innovation. In alignment with trends around the world, the pandemic catapulted Hong Kong's economy into a fast-growing digital economy with various ecosystems embracing digital platforms, tools and tech-enabled solutions, to solve problems and drive innovation. In fact, in 2020, Hong Kong's leadership committed over US $12.9 billion towards innovation and technology development.
These shifts have led to the exponential increase in the amount of data being consumed and produced, and in turn, the demand for talented, savvy data engineers is expected to shoot up by almost 50% in 2021!
One key reason for this upward trend is that almost every ecosystem from financial, healthcare and government, to logistics and travel, is increasingly powered by technology. In such a digitally-connected world, the frameworks that capture, categorise and deconstruct data need to be superior and sophisticated, so that data can be analysed, consumed and leveraged in the fastest, more efficient way.
The genius architects of such frameworks have come to be known as data engineers.
What Data Engineers Do at Work?
A data engineer's role is one of growing relevance across a wide range of sectors and organisations. A data engineer builds from scratch, tests and maintains data architectures. Some common examples of architectures include databases, as well as processing systems that have the capability to absorb and process mammoth amounts of raw data.
Often raw data contains errors and data engineers are accountable for improving the reliability and quality of the data so that it can be used efficiently for a wide number of outcomes.
Data engineers need to be creative, have an eye for detail and be able to use their knowledge of languages and tools, to come up with solutions that can clean up and categorise data in the most consumable, and relevant way. They develop data set processes for the purposes of modelling, mining, and production, and deliver data that is formatted, scalable and most importantly, secure.
These data architectures, in turn, are used by a large number of people - from data scientists and data analysts to business strategists, journalists and policymakers to drive business outcomes, social impact and generate new information. Hence, ensuring that data architectures and processing systems can convert raw data into consumable data in the shortest period of time, is an extremely important step in any ecosystem today.
Is Data Engineering Different from Data Science?
You might be wondering what is the difference in roles when it comes to data engineers vs data scientists vs data analysts?
Though they are closely connected, the purposes of data engineering, science and analysis are quite different. While data engineers build the architectures that host and process large volumes of data, a data scientist's role is to conduct research and experiments on the whole process of data analysis and organisation of data, to make it more efficient and outcome-driven.
Read Also: Data Analytics vs Data Science
Data analysts, on the other hand, analyse the data and leverage it to understand consumer behaviour, trends, hidden patterns and other outcomes, based on what the data says.
For example, if you work for a social network, the data engineer's role is to ensure that the platform can process a large volume of the data that is posted and consumed on the network. However, the data analyst's role is to analyse the content, usage and consumer behaviour. These emerging insights are then leveraged by various stakeholders like product developers, marketing and community teams, to build a superior product.
These roles are complementary, and some data engineers are also able to analyse data and vice-versa. Yet, the role of data scientists has been evangelised in a big way leading to a rise in the number of professionals entering this profession.
Read Also: 10 Ways to Activate Your Data Science Career
Data engineering, on the other hand, has not been evangelised as much, even though it is one of the most important roles in a digitally-bullish world.
The demand for data scientist jobs in Hong Kong, as well as data engineers, has gone up, and the demand will increase further in the next five years.
How is Data Engineering Used in the Industry?
As more ecosystems get digitised, more organisations are setting up robust data engineering teams to build systems that can process mammoth amounts of data passing through them. In 2016, the demand for data engineers had already exceeded supply. The pandemic of 2020, has taken demand to the next level across the world, and especially in a digital economy such as Hong Kong.
For example, the lockdown of 2020 led to the digitisation of several basic services in Hong Kong. The number of consumers switching to digital payments grew. Last-mile delivery for basics such as groceries and food shot up, and e-commerce flourished.
These shifts in consumer behaviour have made businesses in Hong Kong - from fintech and food delivery apps to retailers - bullish in building their data engineering teams to process, maintain and secure the large swathes of data entering their systems. More and more businesses also began adopting cloud services to make data accessible on the go.
Hence, data engineers, today, work across almost every industry and ecosystem. The top five industries include healthcare and pharmaceutical, telecommunications, Internet, energy and automobiles.
A Growing Demand for Data Engineers in Hong Kong
In 2021, Hong Kong will see growth in several sectors including fintech, insurtech, regulatory and compliance in the banking sector, technology and human-centred design. Additionally, the leadership in Hong Kong has committed over US$116 million to provide essential digital infrastructure to accelerate smart city development in the city.
As companies race to digitise their systems, conform to various compliance frameworks and meet consumer demands, the need for experienced, visionary data engineers is slated to be on the rise. Data engineers will play a key role in these transitions.
It's easy to conclude that the world is an oyster for data engineers who have the skills, knowledge and savviness about a fast-moving digitised world. This is not a bubble that will go bust. The data engineer's role has become cemented in the success of any enterprise, and one that is even more lucrative than that of a data scientist, even though the latter is somewhat more hyped.
Data Engineers’ Salaries in Hong Kong
So, what is a typical data engineers' salary in Hong Kong? On average, a data engineer in Hong Kong can command HK$759,687. The average base salary is around HK$387,555, and this figure can vary, based on the skillset, experience and knowledge base of the candidate. For example, data engineers with a knowledge of Python (HK$394,937) and SQL (HK$456,000) tend to command a higher salary than those who know ETL (extract, transform, load).
As we can see, there is a skills hierarchy in the field and investing in learning specific skills can give one an edge. As one grows in experience, one can earn HK$948,031 and above.
On the other hand, the average base data scientists salary in Hong Kong is HK$400,200. Interestingly, as data scientists and engineers grow in experience, the latter is more likely to earn more as they ascend the career hierarchy.
Read Also: How to become a Data Scientist in Hong Kong
Career Options in Data Engineering
Data engineers play a wide range of roles in the industry, based on the kind of organisation and sector they work in.
A vast number of businesses today are looking to solve a pretty universal problem - they seek more affordable, turn-key and scalable solutions to secure, store and process the data entering and exiting their ecosystems.
More and more businesses are moving their data to a cloud so that it is accessible on the go. A data engineer's role is to design the frameworks that lead to this outcome in the most affordable way. These systems also need to be designed in a way that enables the processing of vast volumes of data in the least amount of time. The data needs to be stored in data warehouses in various formats after being cleaned up.
Yet another concern that data engineers are responsible for is to ensure data security. Large volumes of sensitive, confidential data are being generated every nanosecond, which needs to be protected. The more vulnerable it is, the easier it is for hackers and other entities to access and exploit it.
Data engineers typically leverage software programming languages to clean up the data, ensure that it is of consumable quality and that it is available to other teams. There is an entire cycle needed to make this a reality, and this comprises the ideation and architecture design, followed by prototyping and testing, deployment and monitoring the working of any architecture in real-time.
Technical Skills Required for Data Engineers
Data engineers must possess a good mix of skills, from understanding software programming languages to tools that help them design architecture, secure and protect data.
So, what coding languages do data engineers use? Well, there are several skills that overlap with those of data scientists and software engineers. These include:
- Python
- SQL
- Data analysis
- Big data
- Business intelligence
Additionally, they must be experienced in certain unique skills, including big data tools like:
- Hadoop
- Unix
- Databases
- Hive
- Oracle
- ETL
Tools for writing ETL pipelines include:
- Airflow
- Cloud Dataflow
- Kafka
They must also be familiar with various warehouse solutions and the usage of tools like:
- Teradata
- Oracle Exadata
- Amazon Redshift
- Google BigQuery
They must have worked with various data storage technologies and be well-informed with frameworks that can be combined to build data pipelines.
Soft Skills Needed for Data Engineers
While technical skills form the base for success, data engineers must be knowledgeable about the overall business outcomes they must drive. Problem-solving skills, as well as the ability to work in multi-disciplinary teams, gives them an edge in generating quicker solutions. Needless to say, great data engineers are obsessed with data - eating, sleeping and breathing the flow, security and consumability of the data they are accountable for.
Accountability is also a key quality in any data engineer, and one is responsible for the security of data that impacts the lives of millions of people. Such a role is not for the faint-hearted, and one has to "own" this job, from the get-go.
One can take up a holistic data engineering course in Hong Kong to prepare oneself for a career in this field. Some examples include a Bachelor of Engineering in Computer and Data Engineering or a Master's in Big Data Technology.
Do You Need A Tech Background To Be A Data Engineer?
In today's world, the relevance of skills is growing, and in an area like data engineering, skills and acumen trump degrees. Data engineers typically have an undergraduate degree in Maths, Science or a business-related field. They can also come from a software engineering background. In fact, there are many software engineers in data engineering roles across the world.
However, many of the top data engineers are self-taught. Excellence does not come from degrees but a passion for data engineering and its impact on the larger world we live in.
One can learn on the job, or else take up online programmes that teach a multitude of software languages, and data engineering skills. To begin with, one can take up a Python course in Hong Kong. Additionally, it also helps to do a data science course in Hong Kong, as being able to look at engineering from the lens of a data scientist can be useful in the long-term.
One of the key reasons for this is that technology is constantly changing and evolving, and one must have the acumen to pick up new skills and adapt to changes in technology swiftly to stay fresh and ahead of the pack.
Conclusion
Data Engineering in Hong Kong is booming! This is a great time to invest in the skills that can lead to a very interesting and rewarding career. Due to the lack of reputed universities with specialised courses in data engineering, your best option is to complete a graduate or an undergraduate degree and go for a data engineering course offered by independent institutes, like Xccelerate.
If you are at a choice point in your career and need someone to help you navigate professional challenges. You can make an appointment to our complimentary 1-on-1 Career Consultation and receive personalised career advice.
Read Also: Is Data Science Difficult to Learn?
