Bsrgvty

Will Curiosity and the Mars 2020 rover be able to communicate with each other via a Mars orbiter?

Designing Borders with QGIS

I need to draw this picture in LaTeX. How do I do that? Boxes with arrows

Who are the strongest non-professional GMs?

What antenna is this in an Apollo 15 LM photo?

Hough transform algorithm - Idiomatic c++

Can a Barbarian/Wizard multiclass trigger a magic item while raging?

Difference between apply and funcall

Wifi near the speed of light

Large products with glass doors

Strange labels on our servers

What is the contemporary meaning of primary storage?

Patent Agreement in Order to Graduate

Is 'lay lady lay' grammatical?

Could dinosaurs breathe modern air?

Were there ever 12-, 24-, 48-, etc bit processors?

What are these color strips for?

Is military allowed to wear civilian clothes when testifying in Congress?

Lost Time at Motel?

How to save current font so that it can be used again later?

Rats biting off fuel line ( again and again and again )!

Did any 360-compatible machine implement registers in core?

How to handle a colleague who appears helpful in front of manager but doesn't help in private?

Is it possible to protect saved games on Nintendo Switch from being played by other users?

Managing large data files, vector and raster, in QGIS 3

How to repair and clean contours from high data resolution (lidar)?Per-pixel (statistical) calculations on a raster stack using GDALPoor performance with storing large rasters in PostGIS and visualising in QGISExport vector layer to raster in QGISPacking tiled rasters into a Geopackage in QGIS 3.0.3 whilst maintaining the colourWhat does the vector to raster option in QGIS do, exactly?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;

2

1

I work for a local authority so use data covering a large area. Therefore some datasets I use become large and cumbersome.

For example I have a vector layer for contours and a raster layer for hillshade. As I am relatively inexperienced I created Booth of these of a series of tiled points and created a single file. I am happy with the results but performance is slower.

What are the main options available to improve the performance of the vector layers for a single user on one PC and for a team? This should allow me to both display and analyse the data efficiently.

What are the main options available to improve the performance of the raster layer?

qgis data-management

share|improve this question

edited Sep 12 at 20:49

csk

17.1k11 gold badge1515 silver badges4242 bronze badges

asked Sep 11 at 20:31

boberdorfboberdorf

89199 silver badges1414 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Do you have access to a server, so you could host your own WMS? Could you reduce the resolution of the DEM/countour layer? What do you mean by "I created both of these of a series of tiled points and created a single file"? Aren't they two files? Why do you actually need hillshade? Would it be feasible to turn on these layers only when you really need them?
  
  – Erik
  Sep 12 at 9:32
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  To improve performance of a raster layer in qgis, convert it to a single GeoTiff and add external overviews (fastest) or a virtual raster with external overviews (generally fast). You can create a virtual raster then save it as a GeoTiff. Overviews are created in the properties view (right click the layer).
  
  – vinh
  Sep 12 at 15:52
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  You could try creating a geopackage. This uses a sqlite database and can include optimisations for both vector (spatial index) and raster (overviews and internal tiles).
  
  – vinh
  Sep 12 at 15:54
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  In addition to the suggestions to build a spatial index for vectors and pyramids for rasters , you might check out Settings - Options - Rendering for some settings that affect rendering speed. You can generalize/simplify vectors like contours (those from LiDAR are often quite noisy) as well. If you have rasters with complex symbology, especially 32 bit floating point, converting them to rendered 8 bit can greatly improve speed. If you have complex labeling expressions, masking or locations, or complex symbology expressions or symbols, those can slow drawing.
  
  – johns
  Sep 12 at 17:12
  
  

  
  
  
  

add a comment
 | 

2

1

I work for a local authority so use data covering a large area. Therefore some datasets I use become large and cumbersome.

For example I have a vector layer for contours and a raster layer for hillshade. As I am relatively inexperienced I created Booth of these of a series of tiled points and created a single file. I am happy with the results but performance is slower.

What are the main options available to improve the performance of the vector layers for a single user on one PC and for a team? This should allow me to both display and analyse the data efficiently.

What are the main options available to improve the performance of the raster layer?

qgis data-management

share|improve this question

edited Sep 12 at 20:49

csk

17.1k11 gold badge1515 silver badges4242 bronze badges

asked Sep 11 at 20:31

boberdorfboberdorf

89199 silver badges1414 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Do you have access to a server, so you could host your own WMS? Could you reduce the resolution of the DEM/countour layer? What do you mean by "I created both of these of a series of tiled points and created a single file"? Aren't they two files? Why do you actually need hillshade? Would it be feasible to turn on these layers only when you really need them?
  
  – Erik
  Sep 12 at 9:32
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  To improve performance of a raster layer in qgis, convert it to a single GeoTiff and add external overviews (fastest) or a virtual raster with external overviews (generally fast). You can create a virtual raster then save it as a GeoTiff. Overviews are created in the properties view (right click the layer).
  
  – vinh
  Sep 12 at 15:52
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  You could try creating a geopackage. This uses a sqlite database and can include optimisations for both vector (spatial index) and raster (overviews and internal tiles).
  
  – vinh
  Sep 12 at 15:54
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  In addition to the suggestions to build a spatial index for vectors and pyramids for rasters , you might check out Settings - Options - Rendering for some settings that affect rendering speed. You can generalize/simplify vectors like contours (those from LiDAR are often quite noisy) as well. If you have rasters with complex symbology, especially 32 bit floating point, converting them to rendered 8 bit can greatly improve speed. If you have complex labeling expressions, masking or locations, or complex symbology expressions or symbols, those can slow drawing.
  
  – johns
  Sep 12 at 17:12
  
  

  
  
  
  

add a comment
 | 

I work for a local authority so use data covering a large area. Therefore some datasets I use become large and cumbersome.

For example I have a vector layer for contours and a raster layer for hillshade. As I am relatively inexperienced I created Booth of these of a series of tiled points and created a single file. I am happy with the results but performance is slower.

What are the main options available to improve the performance of the vector layers for a single user on one PC and for a team? This should allow me to both display and analyse the data efficiently.

What are the main options available to improve the performance of the raster layer?

qgis data-management

share|improve this question

edited Sep 12 at 20:49

csk

17.1k11 gold badge1515 silver badges4242 bronze badges

asked Sep 11 at 20:31

boberdorfboberdorf

89199 silver badges1414 bronze badges

share|improve this question

edited Sep 12 at 20:49

csk

17.1k11 gold badge1515 silver badges4242 bronze badges

asked Sep 11 at 20:31

boberdorfboberdorf

89199 silver badges1414 bronze badges

share|improve this question

edited Sep 12 at 20:49

csk

17.1k11 gold badge1515 silver badges4242 bronze badges

asked Sep 11 at 20:31

boberdorfboberdorf

89199 silver badges1414 bronze badges

edited Sep 12 at 20:49

csk

17.1k11 gold badge1515 silver badges4242 bronze badges

edited Sep 12 at 20:49

csk

17.1k11 gold badge1515 silver badges4242 bronze badges

asked Sep 11 at 20:31

boberdorfboberdorf

89199 silver badges1414 bronze badges

asked Sep 11 at 20:31

boberdorfboberdorf

89199 silver badges1414 bronze badges

3 Answers
3

active

oldest

votes

5

For your contours, you can

• use simple scale dependent visibility to turn the layer off when zoomed out too far


• build a spatial index
  


• you could also try a rule-based symbology to limit the contour interval shown as you zoom out


• you may need to build an attribute index on the elevation field to help speed up any labelling and rule-based symbology

For your raster, you can

• build raster overviews (pyramids)
  

share|improve this answer

answered Sep 11 at 23:32

user2856user2856

35.3k22 gold badges6565 silver badges117117 bronze badges

add a comment
 | 

4

The best way to manage your large datasets is to store them in a spatial database like PostGIS. The loading speed will be twice faster.

share|improve this answer

edited Sep 16 at 1:58

answered Sep 12 at 4:14

Simon GISSimon GIS

26311 silver badge99 bronze badges

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  PostGIS will also allow you to do very fast spatial operations.
  
  – ragnvald
  Sep 12 at 5:09
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks Simon, I don't know much about databases, so will check it out.
  
  – boberdorf
  Sep 12 at 10:11
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  @boberdorf This answer is a bit of an over generalisation. Where I work, our very large datasets are stored in a remote spatial database (which is tuned for speed) and it's still way quicker to pull the data down to a local filesystem format to do anything with it. A local database may be quicker, but will have overhead in setting up and tuning.
  
  – user2856
  Sep 13 at 0:44
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Geoserver is not a spatial database.
  
  – user2856
  Sep 14 at 2:49
  

  
  
  
  

add a comment
 | 

0

Please check out PostGIS

• https://postgis.net/

Also you can access Postgres databases in QGIS

• https://www.bostongis.com/blog/index.php?/archives/271-New-in-QGIS-3.2-Save-Project-to-PostgreSQL.html




• https://docs.qgis.org/2.8/en/docs/training_manual/databases/db_browser.html

share|improve this answer

answered Sep 12 at 10:41

raajraaj

12977 bronze badges

add a comment
 | 

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "79"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fgis.stackexchange.com%2fquestions%2f335621%2fmanaging-large-data-files-vector-and-raster-in-qgis-3%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

5

For your contours, you can

• use simple scale dependent visibility to turn the layer off when zoomed out too far


• build a spatial index
  


• you could also try a rule-based symbology to limit the contour interval shown as you zoom out


• you may need to build an attribute index on the elevation field to help speed up any labelling and rule-based symbology

For your raster, you can

• build raster overviews (pyramids)
  

share|improve this answer

answered Sep 11 at 23:32

user2856user2856

35.3k22 gold badges6565 silver badges117117 bronze badges

add a comment
 | 

5

For your contours, you can

• use simple scale dependent visibility to turn the layer off when zoomed out too far


• build a spatial index
  


• you could also try a rule-based symbology to limit the contour interval shown as you zoom out


• you may need to build an attribute index on the elevation field to help speed up any labelling and rule-based symbology

For your raster, you can

• build raster overviews (pyramids)
  

share|improve this answer

answered Sep 11 at 23:32

user2856user2856

35.3k22 gold badges6565 silver badges117117 bronze badges

add a comment
 | 

For your contours, you can

• use simple scale dependent visibility to turn the layer off when zoomed out too far


• build a spatial index
  


• you could also try a rule-based symbology to limit the contour interval shown as you zoom out


• you may need to build an attribute index on the elevation field to help speed up any labelling and rule-based symbology

For your raster, you can

• build raster overviews (pyramids)
  

share|improve this answer

answered Sep 11 at 23:32

user2856user2856

35.3k22 gold badges6565 silver badges117117 bronze badges

share|improve this answer

answered Sep 11 at 23:32

user2856user2856

35.3k22 gold badges6565 silver badges117117 bronze badges

answered Sep 11 at 23:32

user2856user2856

35.3k22 gold badges6565 silver badges117117 bronze badges

answered Sep 11 at 23:32

user2856user2856

35.3k22 gold badges6565 silver badges117117 bronze badges

4

The best way to manage your large datasets is to store them in a spatial database like PostGIS. The loading speed will be twice faster.

share|improve this answer

edited Sep 16 at 1:58

answered Sep 12 at 4:14

Simon GISSimon GIS

26311 silver badge99 bronze badges

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  PostGIS will also allow you to do very fast spatial operations.
  
  – ragnvald
  Sep 12 at 5:09
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks Simon, I don't know much about databases, so will check it out.
  
  – boberdorf
  Sep 12 at 10:11
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  @boberdorf This answer is a bit of an over generalisation. Where I work, our very large datasets are stored in a remote spatial database (which is tuned for speed) and it's still way quicker to pull the data down to a local filesystem format to do anything with it. A local database may be quicker, but will have overhead in setting up and tuning.
  
  – user2856
  Sep 13 at 0:44
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Geoserver is not a spatial database.
  
  – user2856
  Sep 14 at 2:49
  

  
  
  
  

add a comment
 | 

4

The best way to manage your large datasets is to store them in a spatial database like PostGIS. The loading speed will be twice faster.

share|improve this answer

edited Sep 16 at 1:58

answered Sep 12 at 4:14

Simon GISSimon GIS

26311 silver badge99 bronze badges

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  PostGIS will also allow you to do very fast spatial operations.
  
  – ragnvald
  Sep 12 at 5:09
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks Simon, I don't know much about databases, so will check it out.
  
  – boberdorf
  Sep 12 at 10:11
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  @boberdorf This answer is a bit of an over generalisation. Where I work, our very large datasets are stored in a remote spatial database (which is tuned for speed) and it's still way quicker to pull the data down to a local filesystem format to do anything with it. A local database may be quicker, but will have overhead in setting up and tuning.
  
  – user2856
  Sep 13 at 0:44
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Geoserver is not a spatial database.
  
  – user2856
  Sep 14 at 2:49
  

  
  
  
  

add a comment
 | 

The best way to manage your large datasets is to store them in a spatial database like PostGIS. The loading speed will be twice faster.

share|improve this answer

edited Sep 16 at 1:58

answered Sep 12 at 4:14

Simon GISSimon GIS

26311 silver badge99 bronze badges

share|improve this answer

edited Sep 16 at 1:58

answered Sep 12 at 4:14

Simon GISSimon GIS

26311 silver badge99 bronze badges

answered Sep 12 at 4:14

Simon GISSimon GIS

26311 silver badge99 bronze badges

answered Sep 12 at 4:14

Simon GISSimon GIS

26311 silver badge99 bronze badges

0

Please check out PostGIS

• https://postgis.net/

Also you can access Postgres databases in QGIS

• https://www.bostongis.com/blog/index.php?/archives/271-New-in-QGIS-3.2-Save-Project-to-PostgreSQL.html




• https://docs.qgis.org/2.8/en/docs/training_manual/databases/db_browser.html

share|improve this answer

answered Sep 12 at 10:41

raajraaj

12977 bronze badges

add a comment
 | 

0

Please check out PostGIS

• https://postgis.net/

Also you can access Postgres databases in QGIS

• https://www.bostongis.com/blog/index.php?/archives/271-New-in-QGIS-3.2-Save-Project-to-PostgreSQL.html




• https://docs.qgis.org/2.8/en/docs/training_manual/databases/db_browser.html

share|improve this answer

answered Sep 12 at 10:41

raajraaj

12977 bronze badges

add a comment
 | 

Please check out PostGIS

• https://postgis.net/

Also you can access Postgres databases in QGIS

• https://www.bostongis.com/blog/index.php?/archives/271-New-in-QGIS-3.2-Save-Project-to-PostgreSQL.html




• https://docs.qgis.org/2.8/en/docs/training_manual/databases/db_browser.html

share|improve this answer

answered Sep 12 at 10:41

raajraaj

12977 bronze badges

share|improve this answer

answered Sep 12 at 10:41

raajraaj

12977 bronze badges

answered Sep 12 at 10:41

raajraaj

12977 bronze badges

answered Sep 12 at 10:41

raajraaj

12977 bronze badges