Until you run out of memory, it's normally not an issue. There is a limit on memory bandwidth, so you cannot quickly swap in and out tons of high resolution textures without streaming issues.
High res texture by itself won't change performance, until you run out of memory, as Zac said. Applying complex math on highres textures in a material in the other hand, definitely changes performance. Using parallax with high res textures is just horribly slow for example. The "problem" with 4k textures is that you always need more than one in a material, usually 3. Memory consumption of the 3 textures would be somewhere around 50mb for this material, if the textures are compressed. You can do a quick math to estimate roughly how many 4k materials could you have with certain vram budgets. You also need to spend some of it on other stuff, not only on textures.
If for whatever reason you disabled mip-mapping on the high res texture it would for sure be a pretty big performance hit if the GPU had to perform a minification-esque lookup.
The Texture Fetch time can affect performance of the GPU when your shader has a very large number of high resolution textures. The texture fetch in very layman's terms is essentially the GPU fetching the texture data from memory cache in order to render it. This should not be confused with streaming texture memory. We're not talking about pulling texture data from disk, we're talking about throughput of textures in VRAM to the GPU. Profiling software like PIX, GPad, or more modern day equivalents will allow you to drill into each draw call and evaluate the texture fetch time.
Having said that, rarely have I encountered GPU performance bottlenecks rooted from texture fetch delay. If you do throw enough high resolution textures at the gpu in a single call, it will cause a delay. Modern layered shaders that are blending between 3 or 4 different diffuse, gloss, spec, ambient occlusion, emissive, and mask textures at resolutions like 4096 will certainly impact that shader's texture fetch performance. There are usually a thousand other problems more significant that must be addressed first before the fetch time becomes the sole culprit to your performance problems, but it's not something I would ignore completely either.
Here's a pertinent snippet: "...meshes and textures are stored in memory that is physically separate from the GPU's shader processors. That means that whenever the GPU needs to access some piece of data, like a texture being fetched by a pixel shader, it needs to retrieve it from memory before it can actually use it as part of its calculations. Memory accesses are analogous to downloading files from the internet. File downloads take a certain amount of time due to the internet connection's bandwidth - the speed at which data can be transferred. That bandwidth is also shared between all downloads - if you can download one file at 6MB/s, two files only download at 3MB/s each. The same is true of memory accesses; index/vertex buffers and textures being accessed by the GPU take time, and must share memory bandwidth. The speeds are obviously much higher than internet connections - on paper the PS4's GPU memory bandwidth is 176GB/s - but the idea is the same. A shader that accesses many textures will rely heavily on having enough bandwidth to transfer all the data it needs in the time it needs it. Shaders programs are executed by the GPU with these restrictions in mind. A shader that needs to access a texture will try to start the transfer as early as possible, then do other unrelated work (for example lighting calculations) and hope that the texture data has arrived from memory by the time it gets to the part of the program that needs it. "
Replies
They do take up 16 times as much memory as a 1k though so you're going to run out a lot quicker
https://www.gamedev.net/articles/programming/graphics/gpu-performance-for-game-artists-r4632/
The Texture Fetch time can affect performance of the GPU when your shader has a very large number of high resolution textures. The texture fetch in very layman's terms is essentially the GPU fetching the texture data from memory cache in order to render it. This should not be confused with streaming texture memory. We're not talking about pulling texture data from disk, we're talking about throughput of textures in VRAM to the GPU. Profiling software like PIX, GPad, or more modern day equivalents will allow you to drill into each draw call and evaluate the texture fetch time.
Having said that, rarely have I encountered GPU performance bottlenecks rooted from texture fetch delay. If you do throw enough high resolution textures at the gpu in a single call, it will cause a delay. Modern layered shaders that are blending between 3 or 4 different diffuse, gloss, spec, ambient occlusion, emissive, and mask textures at resolutions like 4096 will certainly impact that shader's texture fetch performance. There are usually a thousand other problems more significant that must be addressed first before the fetch time becomes the sole culprit to your performance problems, but it's not something I would ignore completely either.
Here's a pertinent snippet:
"...meshes and textures are stored in memory that is physically separate from the GPU's shader processors. That means that whenever the GPU needs to access some piece of data, like a texture being fetched by a pixel shader, it needs to retrieve it from memory before it can actually use it as part of its calculations. Memory accesses are analogous to downloading files from the internet. File downloads take a certain amount of time due to the internet connection's bandwidth - the speed at which data can be transferred. That bandwidth is also shared between all downloads - if you can download one file at 6MB/s, two files only download at 3MB/s each. The same is true of memory accesses; index/vertex buffers and textures being accessed by the GPU take time, and must share memory bandwidth. The speeds are obviously much higher than internet connections - on paper the PS4's GPU memory bandwidth is 176GB/s - but the idea is the same. A shader that accesses many textures will rely heavily on having enough bandwidth to transfer all the data it needs in the time it needs it. Shaders programs are executed by the GPU with these restrictions in mind. A shader that needs to access a texture will try to start the transfer as early as possible, then do other unrelated work (for example lighting calculations) and hope that the texture data has arrived from memory by the time it gets to the part of the program that needs it. "