创新及科技解决方案

解决方案编号

S-0462

解决方案名称

智能網站數據抓取方案

解决方案描述

For businesses of all shape and sizes, whether start-ups or Fortune 100s, scraping the web for data to fuel your market research efforts offers the broadest and most insightful perspective of your industry. Manually acquiring data for market research is a mundane, arduous task - one, fortunately, easily automated by intelligently designed web crawlers.

In this connection, we offer a Website Extraction Solution that converts unstructured website data into structured ready-to-consume data. In this solution, we offer a self-built data automation platform (called DataCanva) that can scrap website information automatically, continuously and effortlessly, and perform various data transformation, and then output structured data ready for consumption through files, API, webhooks.

Our Website Extraction Solution has a number of proprietary technologies to enable data crawling at scale even on difficult sites:

a) Anti-ban: Our technology has strategies to emulate a human visit session to avoid banning.

b) Auto-queuing: While some sites have implementing auto queuing feature when the sites are overloading, our technology will enable the crawlers to queue up in virtual waiting room just like a human.

c) Login: While some sites require a valid credential and some session-related mechanics in order to load more data, our technology work seamlessly in these scenarios.

d) Deep crawling: Our technology does not only target at web pages, but also attachments such as WORD and PDF file. 

e) Natural Language Analysis: Our technology can extract key phrases, key sentences and perform summarisation if needed.

f) Data Change Detection: Our technology extract delta change in data to minimize the data crawling workload and allow timely feedback.

g) Rotational Proxy: Our technology leverages a large pool of IP to decrease latency and improve success rate.

h) Screen capture: our technology saves the screen in PDF file for historical snapshot of the website for future review.

应用领域

运输
财经
卫生
发展
环境
广播
人口
工商业
气象
社会福利
房屋
法律及保安
城市管理
食物
基础设施
康乐及文化
教育
就业及劳工

使用的技术

预测分析
数据分析
人工智能
云端运算
自然语言处理
机器学习
深度学习

使用例子

The Website Extraction Solution is suitable if the below use cases:

a) Market trend analysis
b) Price monitoring (e.g. on major E-commerce websites)
c) Research and development
d) Competitor analysis
e) News/alerts monitoring (i.e. good for compliance monitoring)
f) Profile analysis (i.e. retrieve data to enrich the user/company profile)

公司／机构名称

互聯互動科技有限公司

电邮地址

ivan.ng@livelyimpact.com

电话号码

92761341

网址

http://www.livelyimpact.com/

地址

Unit 541A, 5/F., Core Building 2, No. 1 Science Park West Avenue, Hong Kong Science Park

如政府部门对创新及科技解决方案进行概念验证或验证技术请联系智慧政府创新实验室